(This summarizes a problem mentioned in other discussions.)
A recent enhancement to FortiOS, used in the FortiGate firewall, did not handle NTP DNS changes correctly. When DNS mapping changed FortiGate firewalls sent 10 second duration NTP bursts at rates that could exceed 20,000 requests/second. NTP Pool servers were impacted due to the use of DNS load balancing. Our team monitored three NTP pool servers and detected over 150 FortiGate devices sending NTP bursts.
FortiGate support identified the problem: Bug ID 607015
FortiGate support informed us that FortiOS 6.2.4, released on May 12, 2020, fixed the problem. Operators of the FortiGate firewall must install that software, it is not an automatic upgrade. We don’t know when the updates will be complete. Questions should be directed to FortiGate support.
We recommended that FortiGate apply for a Vendor Zone .
The rate of the bursts is different in different zones. From the zones I’m currently monitoring, the US zone seems to be most impacted with about 13% of NTP requests generated by these broken clients. Over time, we should see how quickly are the clients being updated.
Due to unrelated issues FortiOS 6.2.4 adoption has been insignificant. FortiNet now says 6.2.5, slated for mid-August, will be the recommended upgrade. This is disappointing, to say the least.
Does DC mean “Domain Controller” as in Microsoft Windows or something else?
We would need more details on the client software/configuration to speculate on what happened.
The 80 seconds difference means that one or both of the time sources were inaccurate.
If a difference is repeatable, a smartphone with a GPS/GNSS application can be used as a third time source and then determine whether either NTP time or DC time is within a couple of seconds of GPS time.
It is now 11 months since the FortiGate bug was introduced in 6.2.3. The current software version is 6.2.6. I hope that the unwanted NTP request bursts are decreasing.
However, we’re monitoring a few pool servers and still see substantial bursts.
Quick update. It is now September 2021. The FortiGate problem continues.
It is especially severe in Lithuania, comprising 80-90% of NTP Pool traffic.
I’ve been monitoring New Zealand closely: 15-20% of NTP Pool traffic there over the past year has come from these bursts.
I’ve had little success reaching the administrators of the abusive clients.
There is an IP address bothering us every now and then and I would like to find out if it is still suffering from this bug.
The packets are NTP version 3. Below is a picture of the NTP-packet. The source port never changes. It does look like the FortiGate problem, except it aren’t bursts, but rather a continuing stream of packets (well over 70,000 per second), that lasts for some 12 hours on a row.
Can anyone tell if this looks like FortiGate? Or maybe something else? One would assume the FortiGate-issue should have disappeared by now.
The FortiGate problem is still present, though the number of affected clients is slowly dropping. [It’s been almost two years!] The burst duration is typically 10 seconds or less.
Miroslav recently pointed to a systemd bug . The key indicator seems to be "fractional part wrapping around 232 milliseconds " I’ve seen this bug as have several others.
