Also, are the incoming packets or the outgoing packets triggering the “protection”? Is this a feature (triggered by the inbound packets) that protects your system from an attack coming from the Internet, or is this a protection of the Internet from your system because you are flooding with NTP reply packets some specific external systems (protection triggered by the outgoing packets).
Then, put your money where you get reliable service.
It’s the incoming packets. There seems to be some per-flow rate limiting in the network (I see much higher packet rates on other VPSes) and I suspect when the number of limited flows exceeds a threshold, the whole port is blocked for 15 minutes.
OVH is, or at least was when I checked the last time, the most popular VPS provided in the pool.
They should be blocking only the flow that is problematic. If they block all inbound NTP traffic because of one problematic source IP, then rather OVH provides an easy option for denial of service.
OpenAI told this me for the question “Could you tell me how does OVH’s DDoS protection work?”:
5. Customer Transparency
Notifications and Reports: Customers are typically notified of ongoing DDoS attacks and can access reports detailing the nature of the attack and the mitigation actions taken by OVH.
I believe you have notifications and reports too.
Yes, I get the reports. Not much I can do about it except block the addresses. It’s not just one flow, more like 20. When I blast the server from one address, nothing happens except the rate is reduced.
I looked at the individual addresses during a burst that triggered the block and there are some interesting patterns. It looks like it’s an on/off thing randomly switching between the addresses (each row is one second):
I also never really figured out what exactly was triggering this, as it happened at varying bitrates (I didn’t monitor packet rates at the time). Their DDoS protection seems really antiquated (to put it mildly), as OVH then send messages saying that…
Your IP is used in a NTP Amplification attack. You should update your NTP server, or fix its configuration, so that it won’t respond to ‘monlist’ queries.
You can check your server’s configuration here : https://www.ovh.co.uk/cgi-bin/tools/ntp_security.cgi
Attack detail : 31Kpps/19Mbps
dateTime srcIp:srcPort dstIp:dstPort protocol flags bytes reason
2024.04.23 16:08:04 CEST x.y.z.a:123 x.29.73.188:17328 UDP — 76 ATTACK:NTP
2024.04.23 16:08:04 CEST x.y.z.a:123 x.212.243.33:11748 UDP — 76 ATTACK:NTP
2024.04.23 16:08:04 CEST x.y.z.a:123 x.144.170.6:48843 UDP — 76 ATTACK:NTP
[…]
(Address obfuscation by me.)
Had this been an actual amplification attack, the outbound traffic should have been noticeably higher than the inbound one. Which it wasn’t, rather slightly lower due to packet drops on the server side.
Also, the packet size doesn’t seem like monlist queries (but still could be some trickery).
Then, had they used their own configuration check tool, they would have seen that that isn’t an issue.
And lastly, looking at the actual traffic showed that there was nowhere a mode 6 or other control-type packet in sight when those “NTP Amplification” attacks based on monlist queries supposedly happened…
A pedantic correction:
monlist queries were mode 7 (ntpdc). ntpd removed support for monlist as of 4.2.8’s release in December 2014. The ntp-dev branch removed that support in April 2010, and disabled all mode 7 responses by default in November 2011. The replacement functionality, ntpq -c mrulist, uses a nonce mechanism to ensure the requesting IP address isn’t forged (spoofed), as it was with monlist amplification attacks.
Terminology: The company is Fortinet. One of their products is the FortiGate firewall. I had confirmation from Fortinet of the unfortunate FortiGate behavior, which began in December 2019 with a faulty software release. A correction was issued mid-2020. Hal and I monitored several NTP Pool NTP servers and watches as the FortiGate problem dwindled over the next couple of years. Except in Lithuania.
Some NTP servers located in Europe, especially in the Lithuania zone, still show the same signature seen with the FortiGates. I queried Fortinet about this, but they no longer respond to my emails. I don’t think these are from FortiGate firewalls. My belief is that these are another Fortinet product, possibly a VPN. I could be wrong, but I’ll call these “Fortinet VPNs”
The FortiGate firewalls had static IP addresses, as expected. Many of the Fortinet VPNs appear to have dynamic IP addresses, others have static IP addresses. The top ASNs with abusive clients are 1257, 8764, 13194, and 43627.
At times many Lithuanian NTP clients simultaneously (sometimes over 40!) send bursts to the same NTP server. I haven’t determined a pattern. The simultaneous bursts may trigger the DDoS alerts.
The Lithuanian clients are seen in whole Europe, but some countries have their own prominent local forti* bursters. For example in Germany the top 3 bursters I see are from Germany, in France it’s the top one.
I think it depends on how the clients are configured, e.g. if they use europe.pool.ntp.org or pool.ntp.org. For best results the ntpguard rules should be made in each country separately. I’m collecting data in 12 countries.
Yes, this is very annoying. I made a suggestion to the OVH support to add a verification using their own tool before claiming “amplification attack”, but they didn’t seem to be interested.
Sweet! On Linux, I expect that to go nicely with a hash:ip,port-type ipset, or even just a hash:ip or other type, depending on the desired granularity.
Thanks!
Nice tool thanks!
But I have a clarifying question; in the output there’s an Pv4 address 46.36.88.170, being identified as ‘fortigate’. However, when I check it out, it appears to be a MikroTik device running RouterOS v6.45.9.
How should I interpret this?
I think that means the Fortigate (or a different FortiOS-based device as Steven pointed out) is in a local network behind NAT. A firewall or VPN doesn’t have to have a public address, right? The Mikrotik router could be provided by the ISP and their customers have their own device protecting their network.
I’ve seen several MikroTik devices among the Lithuania bursters.
FortOS in the news:
https://www.akamai.com/blog/security-research/2025-february-fortinet-critical-vulnerabilities
I had a misbehaving client in the US zone today that evaded detection by the detector tool despite reaching peaks of 5000pps and more. I guess the traffic signature does not match those currently known for Fortinet products or timesyncd. The pattern was six distinct bursts per minute, for a 5-minute average rate of near 1Mbit/s continuously ever since it started at about 9AM UTC today. I have a PCAP with a few minutes worth of traffic. As it seems I can’t attach files of that type here, it can be downloaded (hopefully) from here for the next seven days.
I wrote an email to the apparent user of that IP address. Until I hear back from them, I have now blocked that traffic on the infrastructure side as it was more than doubling my usual traffic volume.
I looked at the pcap & agree this is not the Forti* problem. I’ve never seen a similar pattern.
