As writen before i think this is still kind of bad firmware of router, kitchen aid, toaster or what ever 
The interval from different IPs havenât changed and was still at -5
IIRC we had this kind of problem earlier:
It was a 2.8GHz Xeon
Well talk to the right person then.
I think there was some buggy systemd-timesyncd implementation too. If I remember well, the signature was that the packets arrived within a well-defined subsecond time range.
My small VPS seems to survive itsâ load. Thatâs good.
https://www.ntppool.org/scores/178.217.98.201
I should have said ntpq -nc "mrulist sort=avgint" as without the -n option ntpq will spend a lot of time looking up hostnames via reverse DNS, and when successful and the reversed name also forward resolves including the IP, will show only the hostname not the IP address.
You may be right, but that evidence does not rule out a DDoS being the triggering factor. Is the NTP server in question marked âmonitoring onlyâ or is it active in ru.pool.ntp.org?
Even if that is a count from a server active in the .ru Zone, keep in mind that a DDoS can be quite sporadic and relatively short-lived to be effective at knocking all servers out of the pool for many hours thereafter as the monitors are quick to remove and drag the score down to -100 and it takes quite a while to work back up to 10.
Also keep in mind that if the attacker is using a DDoS botnet with hundreds of thousands of devices, each IP address may be sending at a relatively low rate. Botnets that large are sadly common today.
Finally, when talking about ISP NAT, more accurately referred to a CGNAT for Carrier-Grade NAT, each public IPv4 address has only 65,536 (2**16) ports to NAT with. Given each subscriber needs at least dozens if not hundreds of ports to not have problems with the CGNAT, my understanding from reading NANOG is ISPs in North America are putting no more than 256 or 512 subscribers behind each public address, not thousands. 512 subscribers means 128 ports for each subscriber. I do not know if CGNATs typically limit each subscriber to that portion, or allows some clients to go over knowing others will use less.
Yes, I agree that would also fit with the evidence so far, if the buggy device just went on sale or buggy firmware was distributed to existing devices, and each buggy device is polling at line rate (which has been seen before).
If on the other hand the buggy devices are just polling every second if they donât get a response within one second, it would look different than this, where level of a few thousand packets per seconds per IP has been noted.
EDITED: And that assumes the buggy device has hardcoded a *.ru.ntp.org NTP server, or the carnage would be more widespread.
This is one of the systemd-timesyncd bugs.
Most of the buggy FortiGate firewall NTP bursters have been fixed. There are a number of devices centered in Lithuania that may be related to FortiNet VPN systems that still send bursts, primarily in Europe.
Iâm interested in pcap samples of other high rate clients.
And despite that issue being filed by @mlichvar over 4 years ago, itâs still open. That may be because GitHub issues are no longer maintained by timesyncd devs, though, as I recall seeing somewhere that it does now have some baseline rate limiting.
Guys from correctly working zones, could you please provide top 10 querying IPs for 1M packets? I wonder if my numbers are normal or normally there should be 3-5 packets from one IP
I donât know if Singapore can be considered ânormalâ, but:
# time tcpdump -c 1000000 -w sg.pcap inbound and ip and udp and dst port 123
real 1m15.576s
user 0m0.115s
sys 0m0.203s
# tcpdump -nn -r sg.pcap | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head
7712 116.12.235.x
5130 118.201.175.x
2021 103.227.241.x
1922 165.21.92.x
631 203.149.235.x
471 219.74.179.x
337 116.12.191.x
281 61.16.92.x
258 222.90.229.x
254 122.11.164.x
For comparison, similar stats for Finland but with only 100k requests, so scale by 10x in your mind. Otherwise I would have needed to wait more than two hours for 1M.
# time tcpdump -c 100000 -w fi.pcap inbound and ip and udp and dst port 123
real 13m52.573s
user 0m0.019s
sys 0m0.075s
# tcpdump -nn -r fi.pcap | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head
1777 83.148.198.x
718 13.52.88.x
659 194.240.127.x
307 193.184.119.x
297 31.172.158.x
275 87.95.173.x
258 135.181.86.x
245 37.16.124.x
211 46.163.255.x
205 95.217.63.x
Moral of the story: All the zones have their own oddities.
1M packets from TW (netspeed at 512k):
# time tcpdump -c 1000000 -i ppp0 -w tw.pcap inbound and ip and udp and dst port 123
tcpdump: listening on ppp0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
1000000 packets captured
1001075 packets received by filter
0 packets dropped by kernel
real 4m52.757s
user 0m0.881s
sys 0m0.971s
# tcpdump -nn -r tw.pcap | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head
reading from file tw.pcap, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144
1687 60.190.162.22
774 121.60.117.223
675 150.116.118.2
405 113.196.87.130
327 202.55.239.130
309 220.130.137.169
291 59.120.152.34
290 49.88.54.178
266 114.35.106.188
260 210.242.156.60
Zones: @ de europe
Net speed: 3 Gbit IPv4 & IPv6 each
root@ntp-nue:~# time tcpdump -c 1000000 -w de.pcap inbound and ip and udp and dst port 123
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
1000000 packets captured
1001978 packets received by filter
0 packets dropped by kernel
real 6m16.749s
user 0m0.072s
sys 0m0.220s
root@ntp-nue:~# tcpdump -nn -r de.pcap | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head
reading from file de.pcap, link-type EN10MB (Ethernet), snapshot length 262144
10291 51.116.133.x
4609 86.103.211.x
3842 93.239.86.x
3289 91.37.122.x
1617 212.170.27.x
1130 95.91.251.x
1117 217.224.239.X
1088 58.96.31.x
984 89.142.198.x
912 85.216.79.x
Zones: @ ch europe
3 Gbit, IPv4 Switzerland
[root@skitty ~]# time tcpdump -c 1000000 -i eth0 -w ch.pcap inbound and ip and udp and dst port 123
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
1000000 packets captured
1001423 packets received by filter
0 packets dropped by kernel
real 10m20.380s
user 0m0.259s
sys 0m0.232s
[root@skitty ~]# tcpdump -nn -r ch.pcap | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head
reading from file ch.pcap, link-type EN10MB (Ethernet)
32904 217.151.113.140
19079 213.221.157.115
5589 178.174.44.202
4520 128.77.84.204
2897 178.198.94.239
2317 217.193.184.82
2248 212.117.127.224
1536 212.147.17.81
1470 81.13.158.161
1446 16.63.113.150
[root@skitty ~]#
Simmilar to other countries.
Any particular reason given? Your CPU usage seems fairly high. For reference, one server of mine seems to take around 50% of a single CPU core at 30k pps (yesterday at 14:00 UTC). Running tcpdump or similar will take a significant chunk of CPU time as well.
No, that was shitty hosting that in 2024 even does not require passport to register a VPS.
I continue to serve a second VPS which is absolutely white and legal.
@kkursor @NTPman @apuls
There is no any filtering and âban scritptâ wonât help, donât waste your time.
I researched traffic dumps, soooooooo
Pool settings: 512k
Real incoming traffic to server: 10Mbps - 855 Mbps
Real packets per second (pps) server get: variable 30,000 - 1,800,000, Most of them have different source ip.
I think any home connections and hardware will be overloaded with this traffic/pps. Any hosting and VPS also will kick you with such traffic/pps too.
25 percent of packets is ICMP port xxx unreacheble.packets
internet ->server: ntp request from port xxxxx to 123
server -> internet: ntp reply to port xxxxx
internet - > ICMP port xxxx unreachable
sometimes the icmp comes from the host that requested the time, sometimes from the transit hosts. Looks like ntp traffic are dropped (with icmp) on firewalls or intâs a spoofed source of bad NAT. In one munute I get ~900 000 (nine hundred thousand) uniq ip that send icmp to me. Wow.
It doesnât matter whether you make restrictions in the firewall/NAT translations or in the NTP daemon settings. This wonât help you. 1.8 million packets per second are comes to you hardware and will be processed by router and server NIC+CPU. No options, they have already arrived. Even if you drop them in iptables it is still load CPU. Itâs like DDoS.
On my server I have ~40% of softIRQ = network packets processing (+ 60% chrony) CPU load and itâs 100% on all 4-core 2.66Ghz CPU. Even I drop all ICMP on external firewall, server cpu load reaches 100% on ~300 000 queries per second. Even with chrony instance per core + linux tuning + good Intel NIC with hardware network offloading and tuning.
Another server administrator get 100% load at E-2246G (6-core 3.6GHz) in a similar configuration. All other tasks on the server died. HuhâŚ
Over 1 million packets per second just kills CPU, no options.
Also packet rate-limitating (mrulist in ntpd/ntpsec) in the NTP daemon are available by default And itself wonât help either when more than a million different source IP come in per second. Each IP will need a little bit of memory in list and CPU processing. By default mrulist have limited size and you need tune it for million different sourceIPâs per second.
Thatâs need gigabytes of memory and CPU processing. And this will not in any way eliminate the need to process packets that have already arrived to the server. Even complicates processing.
Well, good luck to all of us.
Thanks umike, you raise several good points.
Although I donât run a NTP server in Russia, I took a look at the incoming traffic of my server that serves the China zone. Hereâs a breakdown of the traffic:
NTPv4 68,0%
NTPv3 27,5%
ICMP udp port unreachable 2,9%
ICMP echo request 0,7%
NTPv1 0,46%
NTPv2 0,36%
HTTP 0,05% (essentially requests to / of various *.pool.ntp.org hostnames)
ICMP host unreachable 0,02%
ICMP net unreachable 0,0006%
others 0,04%
I would guess that much of my own "udp port unreachable"s come primarily from bad NAT configurations. Your 25% seems indeed very high.
A counterpoint regarding rate limiting at chrony/ntpd level. If there are lots of packets coming from an address that is not reachable, reducing packets sent to that unreachable address will also decrease the amount of ICMP responses. Although Iâm not sure if this would be significant in your case.
The default memory limit is also a very good point. chrony uses 524288 bytes by default for client access logging, which, according to chrony documentation, is sufficient for monitoring about four thousand clients. I have my clientloglimit set to 200000000 (190 MB), which should be sufficient for around 1.5M clients. I guess I should have mentioned this earlier.
Edit: Chronyâs documentation has been clarified since the last time I read that page. Now it specifies that the ânumber of addresses and timestamps is always a power of 2â. I changed my clientloglimit to 268435456 (256 MB) for 2 million clients.