Rate-limiting / banning traffic to server

Server operators
1

Hi, can someone help me find info on rate-limiting/banning traffic to my server? What is considered polite behavior for an NTP client (packets per second, burst)? thanks :grinning_face:

https://s1.mymilo.top

Currently my chrony config has ratelimit interval 5 burst 16 leak 2 kod 2

In addition my openwrt router has

root@r7000:/magne-stuff# cat /etc/nftables.d/30-ntp-ratelimit.nft                                                                                                                                                                          
set ntp_banlist {                                                                                                                                                                                                                          
    type ipv4_addr                                                                                                                                                                                                                         
    flags dynamic, timeout                                                                                                                                                                                                                 
    timeout 5m                                                                                                                                                                                                                             
}                                                                                                                                                                                                                                          
chain ntp_ratelimit_chain {                                                                                                                                                                                                                
    type filter hook prerouting priority -310; policy accept;                                                                                                                                                                              
    iif "wan" udp dport 123 ip saddr @ntp_banlist counter drop                                                                                                                                                                             
    iif "wan" udp dport 123 \                                                                                                                                                                                                              
        meter ntp_permeter { ip saddr timeout 30s limit rate over 8/second burst 16 packets } \                                                                                                                                            
        add @ntp_banlist { ip saddr } counter drop                                                                                                                                                                                         
}
2

I use this: GitHub - bruncsak/ntpflood-report: Logs IP addresses of hosts generating excessive NTP traffic · GitHub

3

Are you having issues with abusive clients, or load spikes causing trouble, or any other issues? If not, I wouldn’t even bother.

I understand the desire to want to enforce clients to be “polite”, but it’s probably not even going to work. Most clients are “smart” light switches, microwaves, cash registers. In most cases, the clients in those won’t even realize that not getting a response is meant to tell them something. In the worst case, that could make matters worse, because badly designed/buggy clients might instead take that as a trigger to double down on polling. Seen that many times, in some cases even with devastating effect (> 80 Mbit/s traffic, vs. two or three Mbit/s without trying to enforce politeness).

The partipants in this forum have the interest, the time, and the understanding to interpret the message a server might want to send by withholding respones. Owners of most clients usually don’t, or even know that their device is getting time from the Internet.

Also, the risk is high to hit legitimate clients. Even with just NAT on a consumer line, the number of IoT and other devices might have multiple of them hitting the same servers, especially in small zones, let alone when CGNAT is in the picture.

Or as one of the basic principles of the Internet goes, "“be conservative in what you do, be liberal in what you accept from others” (Jon Postel).

4

I use in all my Chrony servers just this:

ratelimit interval 4 burst 8

Nothing more.

However, I do have a banninglist of IP’s that keep hitting my server, I do ban those IP’s after I have checked if they are static. I do that in the router, not on the server.
As banning dynamic is useless and Chrony does take care of that.

It’s not a long list, pretty short as most people do behave. :+1:

5

Since the result may be devastating, it’s better to prevent than to react.

6

I think there is a misunderstanding: The effect of blocking misbehaving clients was devastating. Not the other way round, trying to block clients to prevent devastating effects. Just the contrary, removing the block “normalized” traffic again (as far as what remained is “normal” in that part of the world).

7

It’s an AI-bot. I reported it several times.

8

I disagree: your stated 80 Mbps is pretty devastating to an NTP server.

9

I do ratelimit interval 1 burst 10

I do this for a simple reason that anything more than that many packets is probably abuse and silently dropping packets is how to prevent my server from participating in a dDoS as a traffic reflector. With this setting, about 20% of packets to my server are not replied to, on average, with spikes higher.

kod? No. I don’t want to send more packets. Leak? No. 10 packets per second in a 1 second window, per IP address is enough.

Here’s an instant snapshot of a small subnet that just has NTP servers:

30 second NTP server in 193000 bits/sec, 270 packets/sec

30 second NTP server out 167000 bits/sec, 233 packets/sec

They all use ratelimit interval 1 burst 10

If someone has some advice on why this is a bad idea, or I’m open to being educated.

10

Doesn’t work that way, the ratelimit interval is 2^x seconds.
So if you set it to 1 it means an IP can’t poll faster then every 2 seconds, ergo you don’t have much of a limit.

My setting of 2^4 means they are allowed to come not faster then once every 16 seconds and they are allowed to burst 8 packets the first time.

The ratelimit isn’t seconds, it’s 2 seconds to the power of x.

In short:

ratelimit interval:
1 = 2^1 = 2 seconds
2 = 2^2 = 4 seconds
3 = 2^3 = 8 seconds
and so on.

11

Yes, fully agree that that is devastating, never contested that. That is why I called it devastating in the first place. Thus, the characterization of what 80 Mbps might potentially do to a server was not what the potential misunderstanding is about.

Rather, the misunderstanding might have been about cause and effect.

Usually, blocking of traffic is used in an attempt to reduce traffic. In the case I described, blocking had the opposite effect, it increased traffic, to the point it became devastating.

I.e., just trying to highlight that there is no one-size-fits-all to the topic. A general tendency seems to be to do some rate limiting by default. And it’ll work in many if not most cases, just potentially harming clients, e.g., when they are caught behind CGNAT through no fault of their own. But in other cases, like the one I described, it can have harmful consequences also for the server and the infrastructure traffic blocking is supposed to protect.

If you were agreeing that blocking of traffic by default might be a bad idea in some cases, then apologies, as the misunderstanding then was on my part.

YMMV.

12

Your arguments sound well-reasoned, and I would not per se say it is a bad idea, and it anyhow sounds like a different point of view would have a hard time getting through. My personal view just would be that filtering 20% of traffic at a meagre 270 pps would be like throwing out the baby with the bathwater.

270 pps doesn’t sound like a traffic volume that could do any harm to server or related infrastructure, or cause issues that would warrant taking this kind of action, as compared to how it might negatively affect even legitimate clients. Or if that traffic volume were causing issues, rather than setting rate limits, the proper approach would be to reduce the “netspeed” setting, as that would reduce the traffic without affecting clients.

Regarding protection against misuse in DDoS attacks, that is obviously a concern. But again, for that to become relevant, much higher packet rates would be needed to outweigh the harm to legitimate clients.

I am not even referring to the severely underserved zones on the globe, where even the lowest “netspeed” setting brings a multiple of the rate you describe. Just common zones, such as the somewhat underserved USA, or even the almost overserved* Netherlands can see manifold higher traffic than 270 pps, several thousands of packets per second at higher “netspeed” settings. Should that be filtered down to 233 pps, because everything beyond that surely must be malicious, either directed at the server itself, or using the server as attack vector?

I am not saying that a server should’t take measures to protect itself against potential misuse. But in the sense of previously quoted/paraphrased “be conservative in what you send, be liberal in what you accept”, I think a more nuanced approach, including much higher limits (still proportional to expected benign traffic) would be more helpful. Balancing the minimization of negative effects on legitimate clients with the legitimate desire of a server to protect itself and third parties the server could be leveraged to attack.

* There almost never can be too many servers in a zone.

13

CGNAT is an abuse of the IPv4 standard. Therefore, I treat it as such.

14

I would be slightly less harsh. I consider the NAT (or CGNAT) as a dirty workaround for an intrinsic problem of the IPv4 protocol.

15

Sure, wholeheartedly agree. One could even cringe at “normal” NAT due to it breaking the end-to-end principle that the original success of the Internet and its win over competing technologies was based on.

At the same time, I wonder how punishing users who don’t know what CGNAT is, or even IP (“Internet” is WhatsApp and TikTok and Instagram…) helps to stop that abuse. Whether I like it or not, those are a fact of life, work-arounds for originally mostly imbalanced availability of IPv4 addresses, now actual shortage of them. IPv6 is available, but in many places introducing it is slow due to cost reasons, in others the cost is more of an excuse. Yet again, how does punishing users who are oblivious to those topics help improve that situation?

I personally think that the leverage towards improvement, if any at all, does not warrant the harm it causes for “innocent” users, or that it even brings me any significant benefit, or prevents any actual harm to me. Thus as long as, and to the extent that remains true, I bite the bullet.

But as almost always, YMMV.

16

I agree with you, but most websites are not reachable on IPv6.
And dual-stack doesn’t help making a move either.

As long as IPv4 is dominant, IPv6 will never take over.
People simply do not switch when something works.

I know some don’t like me saying that, but it’s reality.

17

Bas, I mostly agree with you.
Only some precision, it is only my view:

I would say this that way: “As long as IPv4 is working, IPv6 will never take over fully.”
I think, it is time to introduce partial failures (like slowness, or anything else you can imagine) to the IPv4-only end users in order to force them to start complaining.

19

It’s not my responsibility that some people have poor internet service behind CGNAT. CGNAT breaks the IP end to end standard. Too bad.

But that’s a lot of hand wringing for nothing, since most clients query multiple servers, when being blocked by one won’t make a difference.

20

According to Google’s IPv6 Statistics for the last few weeks over each weekend more than 50% of their users used IPv6 to access their sites. Even during the week the lowest points were well over 45%. Even though IPv4 usage is still on average more, I would not call it dominant anymore. And yes, Google’s measurement is only users and not servers, so it isn’t a complete picture of the whole internet.