Given that carrier NAT is a thing, what is a reasonable maximum packets/sec per client source IP for a pool server (specifically this server)?
I’ve been seeing very high-rate bursts from single IPs recently (up to approx 20,000 packets / sec), which seems a fair bit beyond what is reasonable, and I would like to throttle that back somewhat - but I’m not sure how far to clamp down on it. I don’t want to inconvenience any legitimate users, but I would like to cut the top off those spikes.
Since a while I am thinking to fully cut those IP addresses, which are misusing the NTP service provided by the server I am maintaining. They do not merit even one NTP reply packet, full stop. But I am not yet on the mode to make definitive decision on this issue to go ahead and implement it. I am still collecting the anger to acquire enough motivation.
Sorry, totally off topic, but quote of the day! 
I have my firewall rate limit IPs at 6/min + 10 burst… Large scale NATting is just a cheap out, and not too bright considering the bad actions of 1 person could get everyone else blocked as most IDS don’t care what the source IP is.
Realistically I guess it depends on your bandwidth and how much you really care. The standard polling rate starts at 64s and scales up to 1024s (~17 min). Assuming all the clients aren’t hitting at the same low polling interval, even at a low min rate per-IP like mine should be able to handle dozens of NATted clients.
20k/sec from a single IP is either being spoofed or just abuse… Block it and move on with life… If it’s that mission critical to the tens of thousands of clients behind a single IP, they will figure out a more efficient and proper solution.
In recent months several NTP pool operators have seen high rate bursts of NTP requests. Hal, Miroslav and I have been looking at sample packet captures. The bursts originate from IPv4 addresses, last 10 seconds, and are typically 1,000 - 20,000 requests/second. A client bug seems most likely IMHO, but we cannot prove it yet. The bug might be in an operating system, systems manager, IoT device, router, etc.
I’ve attempted contacting the operators of a number of these machines to discover their operating systems, etc. Identifying the owner/operator and getting configuration information has proven very difficult, but investigations continue.
Blocking individual IP addresses would be tedious. We saw ~100 offending IPs during a two-day study on a set of two servers.
Rate-limits seem a better short term solution.
That behavior is exactly what I am seeing @stevesommars. I also did a bit of packet capture - they’re all NTPv4, with a source port of 123. Is that what you’ve been seeing too? Happy to send packet captures your way if that would be useful.
Bandwidth is no concern; I can soak up a lot more traffic than this without issue. My biggest issue is CPU load on the timeserver; it can’t cope with more than about 10kpps of NTP.
Blocking individual IPs is both easy and automated; my border firewall already records any IP that originates a high-rate NTP burst. Either blocking or rate-limiting that list is trivial.
I have contacted the ISPs for three of the problem addresses - in all cases, the IP belonged to a large customer, and the ISP was not seeing spikes in asymmetrical traffic on the customer’s port - which, as I’m currently replying, means the customer is likely originating the traffic. Given that, some sort of misconfigured device seems likely.
Is there any more detail from my end that would be useful to help track down the cause of this traffic?
The source port is typically 123, but we’ve also seen src UDP port=634 a few times.
I’ll contact you off-list to discuss the additional data.
To me it looks like ntpdate running on broken Windows. There is a mix of old and newer ntpdate versions, so I assume it’s not a single broken ntpdate build. Can anyone confirm ntpdate from the Meinberg package works correctly on fully updated Windows 7, 8, 10?
@mlichvar I’m happy to run some tests on those operating systems (will be in VM) if you let me know what you want tested. Note I’ve never had anything to do with Meinberg before.
I’m not sure where people usually get builds of ntp for Windows. As I understand it, there are no Windows builds provided by ntp.org. Meinberg has an installer, which seems to be recommended on Internet.
What we are interested is the number of requests and their timing. Normally, a recent version of ntpdate should send 4 requests to each server with a 2-second spacing. In the packet captures we see that it’s sending a flood of requests for few seconds. Maybe it doesn’t do that always.
If you install wireshark and start a capture on port 123, you should see all NTP requests and responses. On command line, you can try ntpdate pool.ntp.org and see if there is not a flood in the wireshark window.
I’ll give that a shot and see what shows up in the logs.
I’m already running some packet captures for @stevesommars - do you want the link to those?
No, thanks. That probably won’t be useful to me. I already have some captures.
I collected some statistics, I have now one day full packet capture. During that period my server received 75691294 packets from 3031139 different IP addresses. 9.7% of these IP addresses was responsible for 90.3% of the full traffic, or other way around 90.3% of the hosts generated only 9.7% of the full traffic. The topmost sender IP sent 3340403 NTP packets.
What was the IP source? Residential? Commercial? Unknown?
217.11.220.180 . It gives only average 38 packet/sec.
Someone may wish to try this script:
I already have a firewall rule that does this. But thanks for posting it; others may find it useful 
.
Would you post your firewall rule? That is likely more performant than a script.
Sure. I use the following (on Mikrotik RouterOS). Anything that exceeds 1000 packets / sec and a 3000 packet burst gets logged and put in the throttling list for the next 24 hours. Anything in the throttling list gets limited to max 1000 packets / sec with a burst of 250 packets; excess traffic gets dropped. All limits are per-source-IP.
/ip firewall raw
add action=jump chain=prerouting dst-port=123 jump-target=high-rate-ntp/throttle protocol=udp src-address-list=high-rate-ntp/throttle
add action=jump chain=prerouting dst-port=123 jump-target=high-rate-ntp/detect protocol=udp src-address=!103.106.65.219
add action=return chain=high-rate-ntp/detect dst-limit=1000,3000,src-address/10s
add action=add-src-to-address-list address-list=high-rate-ntp/throttle address-list-timeout=1d chain=high-rate-ntp/detect log=yes log-prefix="[high-rate-ntp]" src-address-list=!high-rate-ntp/throttle
add action=return chain=high-rate-ntp/throttle dst-limit=1000,250,src-address/5s
add action=drop chain=high-rate-ntp/throttle
add action=notrack chain=prerouting port=123 protocol=udp
I would set a much lower max rate. For example NIST says to limit requests to one every 4 seconds.https://tf.nist.gov/tf-cgi/servers.cgi
In other news Miroslav Lichvar, Hal Murray and I traced these bursts to commercial firewalls. (NTPman and Steve Gilberd provided valuable data.) The vendor is aware of the problem and said that a fix should be available later in May.