(This summarizes a problem mentioned in other discussions.)
A recent enhancement to FortiOS, used in the FortiGate firewall, did not handle NTP DNS changes correctly. When DNS mapping changed FortiGate firewalls sent 10 second duration NTP bursts at rates that could exceed 20,000 requests/second. NTP Pool servers were impacted due to the use of DNS load balancing. Our team monitored three NTP pool servers and detected over 150 FortiGate devices sending NTP bursts.
FortiGate support identified the problem: Bug ID 607015
FortiGate support informed us that FortiOS 6.2.4, released on May 12, 2020, fixed the problem. Operators of the FortiGate firewall must install that software, it is not an automatic upgrade. We don’t know when the updates will be complete. Questions should be directed to FortiGate support.
We recommended that FortiGate apply for a Vendor Zone .
The rate of the bursts is different in different zones. From the zones I’m currently monitoring, the US zone seems to be most impacted with about 13% of NTP requests generated by these broken clients. Over time, we should see how quickly are the clients being updated.
Due to unrelated issues FortiOS 6.2.4 adoption has been insignificant. FortiNet now says 6.2.5, slated for mid-August, will be the recommended upgrade. This is disappointing, to say the least.
Does DC mean “Domain Controller” as in Microsoft Windows or something else?
We would need more details on the client software/configuration to speculate on what happened.
The 80 seconds difference means that one or both of the time sources were inaccurate.
If a difference is repeatable, a smartphone with a GPS/GNSS application can be used as a third time source and then determine whether either NTP time or DC time is within a couple of seconds of GPS time.
It is now 11 months since the FortiGate bug was introduced in 6.2.3. The current software version is 6.2.6. I hope that the unwanted NTP request bursts are decreasing.
However, we’re monitoring a few pool servers and still see substantial bursts.
Quick update. It is now September 2021. The FortiGate problem continues.
It is especially severe in Lithuania, comprising 80-90% of NTP Pool traffic.
I’ve been monitoring New Zealand closely: 15-20% of NTP Pool traffic there over the past year has come from these bursts.
I’ve had little success reaching the administrators of the abusive clients.
There is an IP address bothering us every now and then and I would like to find out if it is still suffering from this bug.
The packets are NTP version 3. Below is a picture of the NTP-packet. The source port never changes. It does look like the FortiGate problem, except it aren’t bursts, but rather a continuing stream of packets (well over 70,000 per second), that lasts for some 12 hours on a row.
Can anyone tell if this looks like FortiGate? Or maybe something else? One would assume the FortiGate-issue should have disappeared by now.
The FortiGate problem is still present, though the number of affected clients is slowly dropping. [It’s been almost two years!] The burst duration is typically 10 seconds or less.
Miroslav recently pointed to a systemd bug . The key indicator seems to be "fractional part wrapping around 232 milliseconds " I’ve seen this bug as have several others.
My server in Finland has seen traffic peaks every now and then, but I only recently started capturing the traffic to see what is going on. Most of the peaks were caused by traffic from Lithuania (see above) – 10 second bursts with ~2k requests/sec. I think this looks like a FortiGate bug. I’ve sent notices to the appropriate abuse email addresses, but so far I’ve only received one automatic reply that a ticket was created.
To that one, I responded with this:
I’ll amend my report by stating that if you are indeed using an older version of FortiOS, you may be vulnerable to CVE-2024-21762:
CVE-2024-21762: An out-of-bounds write vulnerability in the SSL VPN daemon (sslvpnd) of FortiOS. This flaw allows remote, unauthenticated attackers to execute arbitrary code or commands via specially crafted HTTP requests. Fortinet disclosed this vulnerability on February 8, 2024, and indicated it was “potentially being exploited in the wild.” The U.S. Cybersecurity and Infrastructure Security Agency (CISA) added CVE-2024-21762 to their Known Exploited Vulnerabilities list on February 9, 2024, confirming active exploitation. PSIRT | FortiGuard Labs
It would be in your own best interests to upgrade.
I’ll likely include this information in any future NTP traffic correspondence.
There are likely others as well, but this “remote unauthenticated attackers” caught my eye. Now that I had a closer look at that page, CVE-2021-26109, CVE-2022-0778 and CVE-2022-42475 also sound interesting. The owners of those FortiGate firewalls might not care too much about some extra NTP traffic, but maybe a threat of someone hacking their corporate firewall would be enough to make them do something.
I’ve been seeing the bursts for over five years!
For a while I was in contact with Fortinet who confirmed the nature of the FortiGate bug. It’s been several years since Fortinet released a FortiGate software update. Outside of Europe the NTP bursts have mostly died out. Within Europe there are still many bursts seen, mostly from Lithuania.
The Lithuania traffic was different though. The source addresses varied over time. My guess is that these NTP clients use similar NTP code in another application that hasn’t been corrected. When I tried discussing this with Fortinet, they stopped responding to my emails. To be fair, I have no direct evidence that Fortinet software is at fault.
I suspect there are several hundred affected clients. It is interesting to see that at times many clients send bursts at the same time.
I was tired of seeing the Anti-DDoS outages of my OVH VPSes caused by Fortigates and decided to block the clients at the edge network firewall. I wrote an optimized detector in C and python scripts to prepare and load the firewall rules.
A major issue with the OVH firewall is that it can have only 20 rules, so the blocked networks need to be quite large to get a decent coverage of abusive clients. There are many addresses blocked incorrectly (one port only). I think that is better than having the server completely unreachable for 15 minutes every time when too many Fortigates decide to flood my server.
If anyone would like to try it, here is the source code:
I’m still experimenting with adjustments of the coverage to minimize the number of falsely blocked addresses while still avoiding most of the outages. If there is interest I can publish my current rules.
I’m also not sure how to keep the rules up to date. When the they are active, the detector will not see that traffic. Maybe there could be a system where each rule is disabled for a day, but the processing script would need to be improved to track when is each addressed blocked and adjust the weights in processing accordingly.
It’s somewhere in their network. It cannot be disabled and cannot be adjusted, unless it’s a bare-metal machine or public cloud instance. That’s a recent information from the OVH support after I explained multiple times what exactly is the problem and I got through to the second level.
Several years ago they claimed they adjusted the limits, but it never worked for me.
