NTP (GPSd) : Back to 2002 at Oct 24th?

Server operators
1

[not sure if this was posted before - if it was, please forgive me]

TL,DR : from Oct 24, 2021, some NTP servers could suffer a 19 years backwards time warp and go back in 2002.

This is due to a bug in the “gpsd” daemon. The bug resides in the gpsd_gpstime_resolv() function in gpsd v3.20 to v3.22. It will lead to the function providing a backward time of 1024 weeks (that is about 19 years) on 24 October 2021.

More info :

The bug is fixed in version 3.23 of gpsd that was released in early August.

Interesting background article here:

–
Marco

1 Like
2

Upgraded my gpsd instance from 3.22 to 3.23 this week without noticing the bug… The announcement from gpsd team is lacking important information: the gpsd-announce post did not mention the bug at all, and the NEWS file only described the bug in an easily-to-be-ignored line. This bug deserves more attention.

3

Running 3.17, so no problems to be expected :slight_smile:

4

Thanks for the alert… also running 3.17 (possibly the default on Raspbian?).

1 Like
5

Yes. It is. Raspbian is a bit behind, but that is not a bad thing.

6

if one of our rhel ntp pool is syncing from a GPSS infected upstream server will this affect our ntp downstream servers

7

Indeed, it could cascade down to other servers, potentially even stratum 0 ones with a good gpsd. Probably a good time to add more redundant servers.

1 Like
8

No you can’t, as typical NTP or Chrony use more then 1 source to set correct time.
It will ignore infected servers and stop giving time if it’s unsure.

9

Unless infected servers are the majority peers. And if pool servers stop serving time, they degrade the reliability of the pool.

10

@Bas Friend, can you help check whether pool ntp org or asian pool ntp org is not running in this version? 3.20 to 3.22

11

Generally this would not be possible, as default GPSD settings shall listen on loopback interface and cannot be detected from internet. After all the bug shall be triggered in hours, we will find the result soon…

12

I think some servers could ignore a large jump of its reference clock, falling back to NTP servers, but fail later after restart if the refclock is selected before it is compared to other sources. This depends on the configured polling intervals and iburst.

I guess will we see in the monitoring data some servers changing stratum from 1 to 2, but hopefully the number of servers jumping back to 2002 will be very small.

13

I have some old stats running somewhere for the UK pool. I have the following counts for IPv4 only.

  • Score is >10
  • Not scheduled for deletion
stratum count
1 43
2 92
3 46
4 5
5 1

It will be interesting to see tomorrow if that looks much different.

14

In my logs I see 5 servers that changed stratum from 1 to 2 after the midnight. I don’t see any large offsets. There is a large number of servers that I didn’t get any response from yet, but I think that’s just the well-known NTP rate limiting of the Internet.

15

I’ve only see two hosts in the UK change from stratum 1 to stratum 2.

stratum count
1 41
2 95
3 44
4 6
5 1
16

I found two NTP Pool servers that might have been affected, one in the US and one in the UK.

1 Like
17

I guess that the outcome was that there were some outlier cases, but the gpsd issue didn’t really cause a dent in the pool. Thanks in great part to NTP clock selection algorithms and their implementation in ntpd and chronyd, and to the pool selection mechanism.

1 Like
18

There is no need to check it as servers that give bad time are ignored by default.
And no, you can only check this on the server that give out time.
In general you do not need to worry.

Typical servers use more then 1 source and they check.
I run my stratum 1, but it also compares about 6 other servers, when my GPS is wrong it will use one of the other sources are correct time.

Stratum2 servers are supposed to use 3~6 other servers for correct time too.

It will not matter if a few are not correct, the system will know and ignore them.
By the time it reaches the pool all troubles are already filtered out.

The pool also checks the correct time, else they will not be served.

I run chronyd, that is far more smart then ntpd.

19

Many thanks to @marco.davids for mentioning the issue, and, with plenty of advance notice.

2 Likes