With the international tensions running high and correct time being of the essence for a correct functioning of the internet, i started to wonder what would happen during a large-scale disruption of the GPS signals.
Scenario
Large-scale (multiple country-wide) discuption of the GPS signals in Europe, preventing many NTP Pool servers to correctly receive these signals.
Many of the NTP Pool servers are cheap GPS receivers without any significant “holdover”, leading to significant time skew within a day or so (multiple seconds).
Question
How will the NTP pool cope with such a secnario? In essence: how resilient will the pool be?
More Details Questions
When will the lack of GPS signals and the resulting time skew become a problem for the pool?
Will these servers be kicked out of the pool? When will that happen?
What will be the effect on the remaining servers who are either unaffected by the GPS disruption or have better holdover performance? Will they be overwhelmed? How soon will that happen?
Has this scenario ever occurred and what was the result?
Is there a way to prevent collapse of the pool?
I live in Helsinki and according to that map there should be “high” GPS interference. However, I have not noticed any problems with my stratum 1 NTP server here at home, either in my own stats or in the pool graphs.
I’m not particularly worried about this scenario. There are always backup time sources.
Long answer: Because we use PPS, the PPS will go bad first, then the monitors will see you starting to false-tick (if you have no other ref-servers as backup). Making you drop out of the pool.
But if you use ref-servers, that your NTP-server marks as proper-ticker, your server will stay ONLINE, no matter if your GPS-signal is gone.
So it depends on yourself. As such I always have a few other servers that I trust.
In worst case scenario the pool-servers-that-stay-online get overloaded (what happened to me) and the pool goes down in total.
However, I doubt the pool will go down.
Try yourself, cover your antenna with silverfoil so it receives nothing, see what happens.
It should switch to a good backup-server.
That is what the monitors are for, to prevent you from ticking-false and STAY in the pool.
In the original post i mentioned GPS, but I meant GNSS ofcourse.
In case the remaining servers with good holdover performance will stay in the pool while other drop out, this will increase the load ont hese servers significantly.
The question is: will they remain in the pool while their connectivity (from the monitors’ perspective) is poor. My feeling is that they might not. Will this lead to a knock-on effect and put the entire pool out of action?
I am wondering if assigning a certain ranking/higher importance to the servers having decent holdover performance is something that has been considered before? Will this make the pool more resilient? Heavy reliance on servers with poor holdover performance is a potential hazard.
I guess in your scenario with a widespread GNSS outage we will see what will happen, especially since it may also affect a lot of the monitoring servers.
Worst case could be monitors loosing sync and mass declaring pool servers as false-tickers and removing them from the pool.
I hope we won’t experience that, fortunately we have now more geographical diverse monitors in production, but looking at the numbers of european servers checking my pool servers, I could imagine taking a big hit in scoring and maybe removal of my servers that don’t share the same compromised GNSS view if current time.
That won’t happen, because the monitors have a built-in mechanism for checking that their own clock is fine. If their clock is off by more than 10 ms as determined by queries to a number of reference NTP servers (typically government/university-run servers with atomic clocks and such), the monitor will not be checking the clocks of other NTP servers. The worst that could happen is that some monitors that depend on a local GNSS source stop monitoring the pool NTP servers until the disruption is cleared. But with something like 80 monitors that we have now, some other monitor would take that monitor’s place.
I don’t believe the accuracy of my pool servers would be affected, at least. They primarily get their time from an NTP server run by the Norwegian Metrology Service, but there are also some other upstream servers that are also backed by atomic clocks.
Should a lot of other pool servers go offline, well, my servers could handle a decent increase in load just fine. Though I suspect most of the pool servers operated by my fellow countrymen also rely on the Norwegian Metrology Service’s server, so it would likely be just business as usual for the pool in Norway.
For stratum 1 pool servers using GNSS as their source of time, I would hope they have some reliable upstream NTP servers to fall back on, should their GNSS source fail.
Exactly my thinking. Diversity is key. Maybe that could be a recommendation to server operators, even stratum 1 ones, to include at least some upstreams that don’t rely on GNSS, such as local or regional, or even more remote metrology institutes or similar/equivalent. Or upstreams that are indirectly based on such, e.g., radio-based time dissemination such as DCF77 in Germany, MSF in the UK, or other types. Diversity is key (though as noted before, local metrology institutes in some places are already very prevalent, so regional or even remote sources could be valuable, for backup/resiliency).
Same for the pool itself which currently concentrates its references on too uniform types of upstreams, and sources them from a small subset of regions only.
Also, at least so far, GNSS disturbances/tampering are luckily rather localized/regionalized, and not yet global or even continent-wide.
And I also don’t believe those independent sources will necessarily be impacted, because many people hopefully/typically already use them anyway as part of the mix. So they’re already being queried, independent of whether their time input is being used, or whether in normal operation, another source such as a GNSS-based clock is used for day-to-day operations.
Indeed. Stratum 1 servers just relying on their own GNSS without holdover can be easily spoofed.
Should these (e.g. Pi5 with GPS hat) really deserve to be qualified as stratum 1?
In my opinion they should not be, unless they are have hardware holdover backup (OXCO, Rubidium or Cesium). Unfortunately there is no way of knowing if they have this backup.
By just adding more “stratum 1 Pi5’s” we are deluding ourselves into a false sense of security.
I speak from experience with these cheap GPSDO’s without any proper holdover performance.
Strata are purely related to one metric (among various) of distance from a reference clock, with stratum 1 indicating direct access to a reference clock. Nothing less, nothing more. Certainly no performance requirements in my understanding.
The original NTP specs had detailed prescriptions as to internal workings of NTP entities in general, but not stratum-specific. And in my understanding, the upcoming NTPv5 does away with those as well, leaving it more to individual implementations, specifying only externally visible behavior.
Performance requirements certainly would make sense, but perhaps rather as part of some certification program (like WiFi certification for WLAN products).
Is a Pi+gps really considered a gpsdo? Is the PI’s clock being trimmed to minimize drift?
I have a few ublox LEA-M8F modules(from defunct PTPv2 time servers, IGM1100o to be exact). It contains a TCXO which it disciplines to sync with gnss time. It can also discipline an external oscillator, such as OCXO or atomic. This is what I consider to be a gpsdo.
The disciplining of the system clock is primarily for the users of the local system. It is not relevant for serving time as an NTP server (except that that operation also uses the system clock for certain operations). A stratum 1 server gets its time from the reference clock. Servers at higher strata get their time from their upstream servers, and that is what they serve (“NTP time”). This can be done without the system clock being disciplined by the NTP implementation.
As such, it is not the combination of RPi and GNSS receiver that represents a GNSSDO in my view, but the receiver itself (if there is an onboard oscillator), or the receiver in combination with an external oscillator.
Obviously, the oscillator in cheaper receivers are good enough for short-term operation as needed for the GNSS functionality, but not so good for longer holdover times, so it can be debated whether that would really be considered an GNSSDO. (I don’t know enough in that area to say whether there’s defined criteria beyond the mere combination of an oscillator disciplined by the GNSS signals that make a system a GNSSDO, or not. E.g., whether it needs to fulfil certain performance and/or functional requirements, or needs to be separate from the oscillator the GNSS receiver uses for its GNSS functions.)
I was referring to a Pi + GPS as a Stratum1 time server. Is the PI’s clock being disciplined to minimize drift between pps pulses? Or just corrected on every pps?
Edit: Sorry, I see there was extra info added which wasn’t in the email notification.
I believe that is what typical NTP daemon implementations do.
But that is for the benefit of the local system primarily, I think. I understand the time that the daemon serves to clients is derived directly from the reference clock, likely undergoing similar “disciplining” measures as when the system clock is being disciplined, but internally within the NTP implementation (e.g., smoothing out short-term variations such as quantization errors in the generated PPS signal, or limited resolution in the PPS signal receiver system on the host side).
And I think that that daemon-internal time (NTP time) is used to discipline the system clock. But that is a one-way thing, i.e., the system clock is not the basis for obtaining the time served to clients, and the NTP time is maintained independent of whether system time disciplining is in effect, or not.
Maybe one thing at least seems relevant to me from a functional point of view: The idea is to provide a frequency reference to external systems, e.g., a PPS signal, or a 10MHz or other frequency output.
Thus, while the GNSS-disciplined oscillator that a GNSS receiver uses internally could be considered a GNSSDO, I think it usually would not be called that unless some reference signals are generated from that and exported for use by external systems.
I.e., a drop-in, functionally (i.e., provide a frequency reference), for an ordinary oscillator, except with much higher accuracy.
But the PPS is, as far as I’m aware GPS second signal.
But anyway, the MAIN signal (not accurate) should only give date and and an average time.
The PPS-signal will/should correct this to accurate timing.
In short, the SAT-signal has to be real bad, unusable for anything to make it go that bad.
From what I read, the PPS signal needs only date+hour, nothing more to correct it.
So if providers decide to make the signal go ‘bad’, it won’t help.
But if they do, their own navigation will be totally off track, for what ever reason they decide to do it.
The only problem, being offline is: what signal to trust as being right? When not offline too long, it won’t be a problem. Maybe after months? Even then I doubt it has an impact.
As you basically render SAT’s useless…but then they are useless to anybody and everybody.
I don’t think a normal GPS has a internally disciplined osc by default. There is no need for that for positioning. The LEA-M8F I mentioned specifically has a DXO, as it is made for frequency and time applications, and it is non standard enough to be mentioned as a feature. From the data sheet:
Integral disciplined low phase-noise 30.72 MHz system reference oscillator
This reference frequency is supplied on a pin on the module. Unfortunately 30.72MHz isn’t as useful as 10MHz, but that 30.72MHz is still used to determine the pps.
Also from the datasheet:
The standard oscillator frequency for LEA-M8F receivers is 30.72 MHz. The VCTCXO provides good phase noise and 100 ppb (24 hour all effects) hold-over performance. Hold-over can also be based on external oscillators of greater stability (OCXO).
Remember, all the GNSS systems are military systems. They can block it’s civilian functions, while keeping the encrypted military functions. In war time GNSS’s civilian use can be turned off completely…
Who here remembers when GPS turned off Selective Availability?
