Manage/Auth system down?

It appears the manage service (https://manage.ntppool.org/) is down/unavailable. I’m getting the same behaviour on both the normal and the beta sites. So maybe the problem is actually the authentication system.

The status page (https://status.ntppool.org/) suggests everything is fine. Maybe the status page could check the auth system as well?

The reason being that a user can’t use the admin system if the auth system doesn’t work.

I didn’t know about the status page until I had a hunt around. It looks like ‘something’ happened recently to the monitoring probes but has recovered. But I’m still unable to log in to curse the NJ monitoring station/Zayo.

1 Like

Yes, same for me too on live and beta. We’ll just have to hope that @ask notices and fixes it… :man_shrugging:

The last probe from monewr1 I observed was 2021-05-08 07:00:03 UTC.

I no longer have any access to monewr1 & so cannot investigate.

Ditto. Management link is not working.

Perhaps after this is finished a Management Portal entry can be made on the Status page :joy::joy::joy:

Perhaps after this is finished more than one person should have admin access! :scream: :man_facepalming:

1 Like

Looks like he’s noticed! https://status.ntppool.org/

Indeed I did; apologies for not updating here or on the status page earlier! It was pretty bumpy to figure out what was going on. We kept debugging after having the systems stable and we think we were hit by this issue. (Downgrading runc got everything stable again after a bit of cleanup).

It’s been a long long year (as it’s been for many of you!) so my work has been quietly keeping things running and not much participation here. Upgrading and ending up with a “bad” version of a component in the system earlier in the week was part of the work to get everything back in shape and get back to work improving things. :slight_smile:

1 Like

It’s been quite a year yes! :no_mouth:

Does anyone other than yourself have admin access?

Having trouble resolving the IP address for management pages currently:

DNS_PROBE_FINISHED_NXDOMAIN

root@sam:~# host manage.ntppool.org
Host manage.ntppool.org not found: 2(SERVFAIL)

root@sam:~# host ntppool.org
ntppool.org has address 147.75.38.240

root@sam:~# host manage.ntppool.org 2001:8b0::2021
Using domain server:
Name: 2001:8b0::2021
Address: 2001:8b0::2021#53
Aliases:

Host manage.ntppool.org not found: 2(SERVFAIL)

root@sam:~# host manage.ntppool.org 217.169.20.20
Using domain server:
Name: 217.169.20.20
Address: 217.169.20.20#53
Aliases:

Host manage.ntppool.org not found: 2(SERVFAIL)

root@sam:~# host manage.ntppool.org 9.9.9.9
Using domain server:
Name: 9.9.9.9
Address: 9.9.9.9#53
Aliases:

manage.ntppool.org is an alias for ewrlb.develooper.com.
ewrlb.develooper.com has address 147.75.38.240
Host ewrlb.develooper.com not found: 2(SERVFAIL)

Only Google seems to have it:

root@sam:~# host manage.ntppool.org 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

manage.ntppool.org is an alias for ewrlb.develooper.com.
ewrlb.develooper.com has address 147.75.38.240

Going to https://147.75.38.240/manage gives “default backend - 404” once past the certificate errors.

hello, great job @ask , the improved status page was super helpful, even before the issue was resolved

I had never heard of Ceph file system before this :joy: :joy: :joy: