hey @ask
why is there actually no possibility to report bad prefixes and drop them directly on the DNS servers? I think this would also put a bit more pressure on the providers that they clean up their mess. The current situation is that we (the operators) have to spoon out the providers laziness.
Btw. the attack still exists and generated ~7.5GB of incoming traffic so far.
Complications I see with this:
Can you elaborate as to what IP prefix you would request be banned from the pool’s DNS servers in order to affect this one 1&1 customer that is attacking you? I otherwise can’t think what you would be asking for other than all resolvers of 1&1 to be blocked which would of course deny service to entire of 1&1, and while I have no reason to doubt your story, that’s a big hammer to wield - is the collateral damage justified and if so what mechanisms would be used to prevent errors?
Each report has to be investigated by someone, otherwise a largely anonymous person (pool server account) has the ability to deny pool service to arbitrary number of prefixes, by submitting mistaken oir malicious reports. Who will be doing that investigating?
I currently have no answer for that. But I’m pretty sure that this problem will be solvable somehow.
IPs behaving badly likely do not care about their own NTP service, so by what mechanism is denying them NTP service meant to alert anyone of anything?
It doesn’t really matter whether the IPs are interested in their NTP services or not. At least the operators are protected from such devices/users/whatever. You must not forget that we are talking here about largely volunteer operators who can rightly expect that the resources they provide will be handled with care. Because, after all, they have to pay for it.
What makes you believe that the IP address doing the DNS query (a DNS resolver) is going to be at all related to the IP address doing the NTP query? Is this supposed to parse and believe EDNS client subnet data?
Can you elaborate as to what IP prefix you would request be banned from the pool’s DNS servers in order to affect this one 1&1 customer that is attacking you? I otherwise can’t think what you would be asking for other than all resolvers of 1&1 to be blocked which would of course deny service to entire of 1&1, and while I have no reason to doubt your story, that’s a big hammer to wield - is the collateral damage justified and if so what mechanisms would be used to prevent errors?
jep, Touché. I hadn’t thought of forwarders and DNS Caches → show stopper…
Are you talking about the mode 7 ntpdc -c monlist reflection attack? Given it’s been a decade since that was ripped out of ntpd, I think you can reasonably blame the operators of ancient versions of ntpd rather than the software.
Perhaps it was a mistake on my part to fail to raise a stink in 2010 when I became aware of the possibility and provided an amplification-minimizing alternative (ntpq -c mrulist) and removed the monlist code from ntpd entirely. On one hand, it might have led to more ntpd operators updating their software sooner. On the other hand, we know the tendency of uptime pissing contests to lead to people not applying security updates, and it would have publicized the attack vector when far more vulnerable ntpd instances were live.
Keep in mind that many clients use DNS servers outside their prefixes and even outside their AS. You can’t assume the prefix of the DNS query equates to the prefix of the client.
For example, I configure all systems I manage to use quad9.net, via encrypted DNS when the client supports it.
In my experience as a small pool server operator, such participation increases the attack footprint considerably. Methinks that malicious actors query the pool to obtain lists of potential servers for attacking. Thankfully, most hackers are rather amateurish and I know of no free VPN service that works with IPv6, which makes adding an IPv6 only server to the pool an effective way to mitigate abuses for now.
@ask, what are you doing to help server operators, especially small ones, to mitigate malicious attacks?
However, only v4 clients understand KoD, yet the pool doesn’t allow restricting the service to v4 clients with restrict version because the monitors probe the servers using v3. This just highlights an issue with NTP in general, ntpd in particular and the pool. So hand wringing RFCs of an old standard is not a solution to real problems experienced by server operators and pool clients alike.
This is not true; as I said in previous post, I used restrict version for years, without specific treatment for monitors, and did not get score penalty from this. It is clients’ responsibility to use current protocol, like we will not support SSL anymore just to keep IE6 working.
SSL is insecure (and doesn’t support virtual hosting); is there anything really wrong with old NTP versions?
If you use Chrony it’s quite simple to block such behaviour:
ratelimit interval 2 burst 6
interval interval
This option sets the minimum interval between responses. It is defined as a power of 2 in seconds. The default value is 3 (8 seconds). The minimum value is -19 (524288 packets per second) and the maximum value is 12 (one packet per 4096 seconds). Note that with values below -4 the rate limiting is coarse (responses are allowed in bursts, even if the interval between them is shorter than the specified interval).
burst responses
This option sets the maximum number of responses that can be sent in a burst, temporarily exceeding the limit specified by the interval option. This is useful for clients that make rapid measurements on start (e.g. chronyd with the iburst option). The default value is 8. The minimum value is 1 and the maximum value is 255.
In short, the abuse client will be blocked in seconds from getting responses.
Why do you allow 128 bursts? Normally 6~8 is allowed.
Maybe that client is bursting all the time?
Then this has changed since the last time I checked… years ago. 
Uh wow, i think I found a solution for that kind of bad clients. The problem existed until a few minutes ago. I just heavily bombed the client with random data. After 1-2gib of trash, which sometimes hit valid NTP flags, the client crashed or at least blocked me.
...
23:01:31.652867 IP 217.197.91.176.123 > 83.135.163.1.32896: NTPv0, Reserved, length 1472
23:01:31.652869 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652870 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652871 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652873 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652874 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652885 IP 217.197.91.176.123 > 83.135.163.1.32896: NTPv3, Broadcast, length 1472
23:01:31.652887 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652888 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652890 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652891 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652892 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652904 IP 217.197.91.176.123 > 83.135.163.1.32896: NTPv7, unspecified, length 1472
23:01:31.652906 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652908 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652909 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652910 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652911 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652943 IP 217.197.91.176.123 > 83.135.163.1.32896: NTPv4, Server, length 1472
23:01:31.652944 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652946 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652964 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652946 IP 83.135.163.1 > 217.197.91.176: ICMP 83.135.163.1 udp port 32896 unreachable, length 556
23:01:31.652965 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.652967 IP 217.197.91.176 > 83.135.163.1: ip-proto-17
23:01:31.663784 IP 83.135.163.1 > 217.197.91.176: ICMP 83.135.163.1 udp port 32896 unreachable, length 556
23:01:31.678406 IP 83.135.163.1 > 217.197.91.176: ICMP 83.135.163.1 udp port 32896 unreachable, length 556
23:01:31.678567 IP 83.135.163.1 > 217.197.91.176: ICMP 83.135.163.1 udp port 32896 unreachable, length 556
23:01:31.686901 IP 83.135.163.1 > 217.197.91.176: ICMP 83.135.163.1 udp port 32896 unreachable, length 556
It sounds like your solution to a misbehaving client is a seriously misbehaving server. It feels like an invitation to more trouble to me.
Now, this is a new form of DoS attack against a server just added to the pool:
$ ntpq -c mrulist
Ctrl-C will stop MRU retrieval and display partial results.
lstint avgint rstr r m v count score drop rport remote address
=====================================================================
0 1.60 0 . 6 3 6 0.251 0 46485 localhost
...
37 35 20c8 . 3 4 17 0.052 0 123 2600:1700:e640:9e30:5a67:7fff:fe9a:34a0
41 29 20c8 . 3 4 20 0.052 0 123 2600:1700:d87a:aa50:5a67:7fff:fe9a:58ca
42 35 20c8 . 3 4 17 0.052 0 123 2600:1700:1113:bbd0:5a67:7fff:fe9a:4864
42 39 20c8 . 3 4 15 0.052 0 123 2600:1700:13e8:8040:ba27:ebff:fe9c:4a1a
43 33 20c8 . 3 4 18 0.052 0 123 2600:1700:d87a:1130:5a67:7fff:fe9a:3caa
43 35 20c8 . 3 4 17 0.052 0 123 2600:1700:1041:7720:5a67:7fff:fe9a:43f7
...
1472 0.060 20c8 . 3 4 4 0.199 0 54701 2600:1004:b092:a2b6:9058:49ff:fe52:2fde
1477 0.060 20c8 . 3 4 4 0.199 0 42718 2600:1004:b0c9:87a5:9058:49ff:fe52:2fde
1493 0.083 20c8 . 3 4 4 0.199 0 41630 2600:1004:b270:700b:d295:c7ff:fe41:c2c4
1498 0.066 20c8 . 3 4 4 0.199 0 58252 2600:1004:b0a8:4d78:2422:39ff:febc:d58
1508 0.068 20c8 . 3 4 4 0.199 0 51683 2600:1004:b261:be69:2422:39ff:febc:d58
1517 0.068 20c8 . 3 4 4 0.199 0 50194 2600:1004:b08d:2201:9058:49ff:fe52:2fde
...
What are the odds of addresses with different networks between /32 and /64 having the same or similar host addresses to the right of /64? The AS [2600:1004::/32] belongs to Verizon, a major retail and wholesale access provider. Could it be doing something funny with IPv6 addresses? Or is it some script kid having fun with the delegated IPv6 address space? Maybe I’ll have to scratch out my remarks above that IPv6 servers are safer in the pool.
It’s not the Server which is misbehaving, it’s just a random data stream done with ncat.
But the point is that the broken client is beaten with his own weapons. If the client crashes, briliant! hopefully someone will check whats happening there (or best case: someone is doing a update and prevent future misbehavior - which is good for all operators!).
So why do you think that defending sounds like more trouble?
Sending a significant amount of garbage to prevent someone else’s systems from working sounds rather malicious. But sounds you’re having fun 
I’m aware of that. If it’s not the server, it’s the server operator misbehaving. I apologize for my imprecise language.
So the client was blasting your server at line rate with garbage sent to UDP 123? I had the impression the abuse involved legitimate NTP queries, am I mistaken?
Even if the client is blasting you with garbage to UDP 123 at line rate, doing the same to them is not defending. Unless, of course, you consider that what they did was not attacking. Assuming you did, you are attacking back, which is not defending. Is English your first language?
Ok, so what’s the alternative? Accept und pay the bills?
It’s not.
How much of a bill are we talking about here?
If a client causes you 1 GB of traffic per day (calculated based on your earlier reaponses), that would be 30 GB per month which is far below all traffic limits I ever heard about.
75 requests per second also should not register as a significant load.
If you somehow pay per traffic or are limited to a point where 30 GB matter I would advise you to move your server somewhere where that is not the case.
@erayd linked a post with a firewall config that blocks high traffic clients, that could be altered to fit your use case if the load on your ntp server is a problem.
For me it sounds like accepting the situation and deploying additional measures that block excessive requests are both perfectly valid alternatives, especially compared to DoSing some random device out in the internet that happened to get your attention.
To be perfectly honest, if one of my servers had a client constantly bombarding it at 75 req/s I wouldn’t even notice.