[chrony] PPS or system clock glitch/spike

[fox]

Hello,
I’ve recently joined ntppool.org with 1 IPv4 and 1IPv6 IP.

I’m using chrony (version 4.9), and I’ve to say I’m quite impressed by the quality of this software!
The documentation also is of excellent quality.

I was also quite impressed by the ntppool infrastructure, especially the monitor system which really is a great piece of distributed system engineering! Congrats guys!

The problem

I’m having regularly (approximately once per hour) a big “glitch” in a lot of metrics that I’m recording using chrony_exporter with Prometheus. Those metrics concern the PPS source and the system clock.

I create this topic in case someone can help me solve this mystery.

The problem also occurs when there is no network traffic and even before I joined the pool.

In the 2 first graphs, NMEA (green) uses the scale on the left of the graph and PPS (yellow) uses the scale on the right.
Look how the standard deviation of PPS from the sourcestats command (first graph) has sudden spikes.
The last offset of PPS from the sources command (second graph) as well.

You can see below them the DOP of my GNSS during that time (I found no correlation between PPS spike and TDOP to be honest).

Here, you can also see the tracking performance which also show glitches at the exact same moment :

The setup

I have following setup:

• Raspberry Pi 2B
• Uputronics GPS/RTC Hat for Raspberry Pi (U-blox Neo M8 + RV3028 RTC with supercaps)

Note: I know that the 2B is far from the best hardware to run a stratum 1.
(To cite a few examples of limitations: IRQs can not be moved and are all stuck on CPU0, the network card is connected to a USB hub, etc.)
But no matter the hardware, I think every hardware can be tuned to try to extract its best performance. But obviously, with the rpi2, I know I won’t be able to go as far as other setups.

I’m using my own Linux distribution, with upstream/mainline kernel 7.2.x (from kernel.org):

• I compile a minimal kernel, flag by flag, and only turn on a feature or driver if I need it
• This machine only runs gpsd, chrony, and node_exporter, that I compiled myself. Other processes are the kernel threads.
• The filesystem is in RAM. Every file in the “/” hierarchy is living in RAM. There is no sdcard driver. (The kernel has an initramfs whose goal is not to mount a root, but to be the OS root itself.)

=> This simplifies the diagnostic and decreases uncontrolled noise compared to normal distributions: no logrotate, no crontab, no systemd, no sdcard I/O etc.
Nothing that I’m unaware of can run on the system.

refclock PPS /dev/pps0 refid PPS lock NMEA poll 0 precision 3e-8
refclock SHM 0 refid NMEA poll 0 precision 1e-1 offset 0.035

rtcsync
makestep 60 1

dumpdir /run/chrony/dump
driftfile /run/chrony/drift interval 300
leapseclist /run/chrony/leap-seconds.list

# 4194304 clients
clientloglimit 536870912

ratelimit interval x burst y leak z

allow

bindcmdaddress 0.0.0.0
cmdallow 10.0.0.1
opencommands activity clients serverstats sources sourcestats tracking

sched_priority 99

gpsd is run like this: gpsd -D 1 -G -n -N -p /dev/gnss0
chrony is run like this: chronyd -n -r

Things I’ve tried

• Use cpufreq governor performance to force CPU to 900Mhz (instead of staying stuck at 600Mhz).
• Fine tune the GNSS to stop SBAS, enable stationary mode, and only enable Galileo and GPS
• Move the GNSS antenna from inside the garage to the roof outside (average Signal to Noise Ratio increased from 20dBHz to 36dBHz, average TDOP decreased from 1.23 to 0.72, used satellites increased from 13 to 16).
• Logged each ublox binary message during a glitch (the lock and time related flags were always stable according to the GNSS receiver)
• Stop the fridge in case the compressor start-up would create an electrical surge?
• Decrease the precision directive from 3e-8 to 1e-6 on my PPS source in case over-estimating it could cause chrony to “over-react”?
• Use sched_priority directive to put the chrony process into SCHED_FIFO mode.
• Lock chrony process on CPU1, 2 and 3 (exclude CPU0 which has all USB, NET_RX, GPIO, I2C and UART interrupts).

The only thing that has “worked”

The only thing that has worked, is use the chrony directive maxupdateskew 1.
It then completely removes all big oscillations.
But as I understand it, it’s just filtering a problem to prevent it from contaminating the system clock. It’s not really removing the original problem.
If I understand it correctly, the directive asks chrony “please close your eyes when you become uncertain, don’t touch the system clock, and then open your eyes again when the storm has passed”.

How to proceed now?

It’s very difficult for me to pinpoint which part of the pipeline exactly has a problem.
It could be:

• The GNSS firmware being unhappy about signal quality
• The GNSS firmware becoming unstable (may be because of a bad configuration or because it is routinely doing something disturbing the PPS signal)
• The capture by the kernel (pps-gpio) of the PPS signal could be delayed due to interrupt latency
• Chrony itself could be woken up late
• The system clock itself could be of bad quality and have spikes

As far as I understand, in most metrics, chrony is always comparing something with the system clock. So it’s hard to know if it’s the source or the system clock which is faulty, of it both are correct and it’s chrony which is messing with the system clock.

Does anyone have an idea on how to debug this?

[Bas]

In my opinion you poll your GPS too often.
You should not poll it that much.
My RS232 GPS is set this way:

refclock SHM 0 refid GPS poll 3 offset 0.083 delay 0.2
refclock SHM 1 refid PPS poll 3 precision 1e-9 maxlockage 32 lock GPS prefer

I found SHM to work better then /dev/pps0, as both signals are generated by GPSD anyway.

Further reduce clientloglimit to about 8192000, yours contains probably way too much entries for such a small system. It will take to much cpu-cycles to maintain them.

Also, your makestep is far to agressive, typical is this: makestep 1.0 3

As for allow, I doubt that does anything as it should be: allow all

And I use these to make it less spiking:

# Make scheduler change
sched_priority 40

# Run only in RAM
lock_all

Hopefully it works better.

[mlichvar]

Increasing poll to at least 2 will enable the refclock-specific median filter, which should deal with corrupted samples if they don’t happen too frequently.
It would be interesting to see a graph of raw offsets logged in the refclocks.log.

[fox]

Thank you very much for your replies!

@mlichvar
I’ve increased the poll interval to “2”, and it indeed stabilized a lot thanks to the median.

However, I’m surprised that the linear regression regularly truncates a lot of points. It that normal?
In my understanding, the linear regression happens after the median filter. Normally, with poll 2, only cleaner points should make it to the regression window.
I often have NR = NP/2, so it looks clean. But very often, NR is 64, and it then suddenly drops to 8.

I’ve made graphs of the raw PPS offset (measured), with the old configuration of poll 0.
The red rectangles show where I’m going to zoom for the 2 other images below.

Feel free if you need more data to help understanding the issue. I also recorded selection, statistics, tracking.
I’m still unsure if it’s the original electrical signal sent by the GNSS which has glitches, or if it’s interrupt latency when it enter linux.
Those graphs have been recorded with shed_priority 1 and cgroup cpuset feature activated:

• one CPU core is fully isolated and dedicated to chrony, even the Linux scheduler doesn’t run on it and nothing else than chrony is allowed to run on this core.
• the CPU core handling all interrupts (that cannot be moved) is also isolated in order to do nothing else than processing interrupts (PPS, UART, USB, NET_RX, I2C for the RTC)
• the 2 last CPUs run the rest (kernel threads, RCUs, and any other application like gpsd).

Whole period (from 22h00 to 09h00)

Calm period before glitch (from 02h15 to 03h00)

Biggest glitch details (from 03h00 to 03h15)

@Bas :

• gpsd doesn’t generate the PPS signal. The pps signal is an electrical pulse sent to a GPIO pin, generated by my GNSS. It’s the kernel with the pps-gpio driver which processes the timestamp in an interrupt handler. Then, it makes it available for any application that reads /dev/pps0 (the application could be gpsd or chrony). In my setup it’s chrony which is reading it directly.
  Do you guys think that we get better precision if we let gpsd associate NMEA with PPS by himself, instead of relying on chrony to do the lockage?

• makestep: I only need it to be able to correct following situation: if the system is powered down for more than 3 months, the super capacitors will be empty, and the RTC will forget the time.
  If the system boots after this situation, the system time starts on 1970-01-01.
  Thanks to the fact that I didn’t put noselect on the NMEA source, the system is able to quickly get out of this situation: the NMEA source will first get selected quickly (PPS is not ready yet and will take time), and the system time is first stepped by chrony to that coarse NMEA time.
  Once this initial stepping has taken place, I never ever want the system time to get stepped again. When PPS will come online, the 2 sources will not fight together thanks to the indicated precision difference between the 2 sources. The error could still be of several hundreds of ms when PPS comes online, but from that moment on, I only want chrony to use slewing to correct the system clock.

• allow directive doesn’t need the keyword “all”. “all” is an optional keyword, as indicated by the brackets in the documentation. It allows all IPv4 and all IPv6 addresses.
  chrony – chrony.conf(5)
  allow all is especially useful to disregard any previous configuration about a given CIDR and override it (for example when using chronyc to change the settings at runtime).

• lock_all is useful in a lot of configurations, but not in mine: in my situation, everything is always “locked”. chrony cannot get out of RAM, because I have no swap.
  And chrony source code (binary, librairies etc) cannot be paged out either because my whole root filesystem entirely lives in RAM, so it’s always in RAM, because there is no other place it could be (there is no storage driver, even the block layer of the kernel is not compiled in).

[stevesommars]

Try disabling chrony & let the CPU drift for several hours. Record the PPS values using ppstest.
Graph the results. For example:

# ppstest /dev/pps0

trying PPS source “/dev/pps0”
found PPS source “/dev/pps0”
ok, found 1 source(s), now start fetching data…
source 0 - assert 1791638274.000000097, sequence: 1536856 - clear 0.000000000, sequence: 0
source 0 - assert 1791638275.000000635, sequence: 1536857 - clear 0.000000000, sequence: 0
source 0 - assert 1791638276.000000099, sequence: 1536858 - clear 0.000000000, sequence: 0
source 0 - assert 1791638277.000000230, sequence: 1536859 - clear 0.000000000, sequence: 0
source 0 - assert 1791638278.000000472, sequence: 1536860 - clear 0.000000000, sequence: 0

Make a plot. x axis=integer part of time stamp. y axis=fractional part of time stamp.
(If fractional timestamp < 0.1, add one. This centers the plot at y=1)

There will be drift (CPU is freerunning). If there are spikes in the plot, something is wrong with PPS.

When you say there is no network traffic, have you checked for unexpected NTP (or TIME or DAYTIME) packets? timesyncd has caused quite a bit of confusion.

[Bas]

@fox

• gpsd doesn’t generate the PPS signal. The pps signal is an electrical pulse sent to a GPIO pin, generated by my GNSS. It’s the kernel with the pps-gpio driver which processes the timestamp in an interrupt handler. Then, it makes it available for any application that reads /dev/pps0 (the application could be gpsd or chrony). In my setup it’s chrony which is reading it directly.
  Do you guys think that we get better precision if we let gpsd associate NMEA with PPS by himself, instead of relying on chrony to do the lockage?

I meant, GPSD also creates a doorway to the PPS signal via SHM.
Mine signals RS-232 DCD and GPSD can process that and turn it into other ways that Chrony and NTPD can understand.
I found the GPSD way to be more stable when fed to Chrony.

As for your glitches, running those Pi’s at full speed is not a good idea.
The reason is they are thermal protected and run the cores down if they get too hot and you can’t control that.
I would rather set it as performance but then at 600MHz however set the upper and lowerlimit the same in cpufreq. As even with performance, it will jump speeds.
Performance alone isn’t enough and at 900MHz all the time it will probably throttle core-speeds to cooldown.

As for the antenna (it’s dB not dBHz) a value of 36dB is very good, but you only need about 28-30dB to function properly, you are well above it.
The number of satellites has little impact on the time-signal, you only need a few and a 3D-fix.

[ccb056]

You’ve got some good Prometheus logs

I’d add CPU usage and temperature, see if there’s any relation to these glitches