Does anyone have details about this or now something? It seems to have started at around 16:10 GMT+1.
Hi DanielRuf. I don’t see a problem from the status page output you showed. The quirk of the line graph including a spurious 0 data point at the end occurs with several of the graphs.
I’m also not seeing any problem looking up *.pool.ntp.org. What problem did you notice?
Our monitoring started telling us at around 16:10 GMT+1, that the local time of our server is out of sync.
The status page messages at the top are normally manually managed in Atlassian. So far I did not check in detail, what the NTP service daemon on the server says.
I will maybe check in detail. So according to the information in the reply, the graph shows no problems.
There are a few thousand NTP servers in the pool. It would help to know at least your country.
Yes, checking your ntp daemon’s status would be a good idea.
1 Like
According to the journallog (checked with journalctl -u systemd-timesyncd), there seem to be timeouts with the NTP server connection. But this does not seem to be new. New seems to be, that the system clock drifts off about 1 minute or more.
I think this may be related to issues on the host, where the virtual machine runs on (my current assumptions are that either the system clock has issues or the CPU is busy with other tasks).
Knowing, like, the IP address of the NTP server your server tries connecting to might help a bit. I think “systemctl status systemd-timesyncd” shows the IP address of the NTP server. “timedatectl show-timesync” may also show something useful.
Besides that, it’s possible that your virtual server gets migrated between different hosts, and the clocks on those different hosts are not set to the same time. It’s even possible that your virtual server can’t set the clock by itself, but it’s locked to the same clock as on the host server. In this case it’s useless to run an NTP daemon in the virtual server.
If you are not running the host servers yourself, I would suggest asking the company running the host servers if they have an idea of what’s going on.
1 Like
It seems like it is related to blocking port 123 in both directions, since even more servers are affected now (at least this is my current assumption, I do not know what was exactly configured in the firewalls).
I do not know much about NTP, so I’m wondering why it started now to be an issue, the relevant firewall change was some months or weeks ago.
Ntp uses UDP on port 123. Blocking that port will block any ntp traffic.
When ntp stops synchronizing, the local clocks of your servers will start drifting away from “true” time. How fast this drift happens depends on your hardware. So it might just have taken your servers some weeks to drift far enough to trigger the monitoring.
If that is indeed the problem, here are some options:
- Allow outgoing UDP traffic targeting port 123 in your firewall if you have a stateful firewall that handles the return traffic
- Allow UDP traffic targeting port 123 in both directions in your firewall
- Set up your own ntp server(s) in your network and configure all machines in the network to synchronize with those
- Then either allow ntp trafic from and to those ntp servers to traverse the firewall
- Or install and configure a non-ntp time source