Puzzling Rate Limiting for Some Monitors

Server operators
,
1

Hi all,

I’ve been facing a puzzling issue with rate limiting for which i hope you could assist in understanding and fixing.

I run a number of NTP servers (IPv6-only) in the pool on various devices.
However, some monitors get rate-limited on some of the devices and not on the other while having identical NTPd/netowrk configuration, resulting in low scores for some. How come?

An example of one of the servers: 2a02:a46d:xxx being affected.
An example of one the servers: 2a02:a46d:xxx NOT being affected while being contacted by the same monitor.

Example of monitors getting rate-limited on 2a02:a46d:xxx: 1783062274,2026-07-03 07:04:34,-3.5,-10.448702812,336,nlams1-1x7260p,5.759,3,RATE
1783061050,2026-07-03 06:44:10,-3.5,-7.314424038,336,nlams1-1x7260p,5.7,3,RATE

It appears to happen only with some monitors from Netherlands or Germany.
Most other monitors have no issue. BTW: I limit access to the servers from some countries in Asia.

Checks i’ve done so far:

  • Checked network settngs for these servers
  • Checked router settings
  • Checked NTP settings. They are currently not set to very restrictive ratelimiting, but just the ntpd default:
    restrict default nomodify notrap noquery nopeer limited kod
    restrict -6 default nomodify notrap noquery nopeer limited kod
    mru initmem 4096

Why are default settings in ntpd causing problems with some monitors?

2

Have you tried to remove temporarily all the rate limiting on the server that has its monitoring in trouble?

3

One potential reason could be that there also is a server behind the same IP address as the monitor, and that that server has your server configured as upstream. This can happen either explicitly (I once statically configured an upstream server that my server was also monitoring, and I got the occasional RATE KoD from the upstream when monitor and server happened to poll around the same time), or indirectly (e.g., when the monitor/downstream server has the Pool itself configured as upstream, and thus randomly getting assigned some of your servers).

As @NTPman suggests, temporarily remove any rate limiting to see whether that improves the situation.

Or generally be a bit more liberal in what you accept by increasing the rate limit beyond the current setting. You only refer to this in context of your IPv6-only servers, but on the IPv4 side, something similar could hit legitimate IPv4 clients that are unfortunate enough to sit behind CGNAT, and to thus share the same public IPv4 address.

4

Thanks for these ideas.
@MagicNTP sounds like a possibility that monitor as well as NTP client are running from same IP/vantage point. For now I want to tryout the idea of @NTPman. Ive turned off rate limiting on one of the servers associated with the following IPs:
2a02:a46d:7ac:xxx
2a02:a46d:7ac:xxx

Let’s see what happens in the coming days.

5

You could also try Chrony and see it it makes a difference.

6

After disabling the ratelimits for a few days the “RATE” as well as the “i/o timeout” for the affected servers disappear, so it clearly indicates a relation.

However, what i am still confused about is that most other monitors do not seem to have a problem with the default NTPd ratelimit settings.
This leads me to believe that the cause of the issue is with the fact that the monitor shares the IP with a server which is also requesting time.
Is this a desired or even an acceptable setup?

7

I don’t think it is a desired setup, but certainly an acceptable one, not least for practical reasons. The monitor also needs to get its time from somewhere. And if a server in the pool is providing good time, not sure it would make sense for the monitor not to be allowed to use it. In some zones, there even might not be that many alternative high quality sources to begin with. And unless the operator knowingly picks the server via the pool, not sure they should be required to find out whether some of their upstreams hopefully configured manually are part of the pool, and to then exclude them.

I personally think that the rather low and strict and rigid default limits enforced by ntpd are hailing from a different, in some ways arguably simpler era. One without CGNAT, and where it was simpler to tell “good” from “evil”. But also one where available resources, both CPU and network, were much scarcer so that a rather rigid enforcement of rather tight limits was more of a necessity than it arguably is today.

8

I’m writing this only to show that the scenario presented above is indeed possible.

I run a few monitor nodes myself. One of those gets its time from GPS, but I wanted to add my Finnish ISP’s NTP server as a backup, so I added those addresses (IPv4+IPv6) to my chrony configuration.

The ISP’s NTP server is also in the pool. After I had configured my chrony to use the ISP’s NTP server as a backup, my monitor started reporting occasional RATE results for the ISP’s NTP server. I removed the ISP’s NTP server from chrony and the RATE errors disappeared. chrony used the default query rate for the ISP’s NTP server and I’m not behind CGNAT.

I don’t know which NTP software they use, but apparently they’ve configured their software’s query limits so that they tolerate the query rates from pool monitors but not more than that.

9

A few of my servers run NTP-server and monitor.
But I have the max-netspeed low enough not to affect the monitor.

Maybe your rate-limit is set too low so it blocks too fast?

Mine is set in Chrony:

ratelimit interval 4 burst 8

Also beware, some firewall-software may see too many requests from 1 network as being an attack and block the port.

Normal servers/clients should not be blocked by this limit.

Hopefully this helps.

10

Yes, there’s also the firewall. Not in this case, because the monitor is receiving RATE.

Some firewalls will block a whole /24 v4 network and a whole /64 v6 one by default. I’ve seen a DNS server pick up my server and then I see activity by a wave of clients from the same network.

This might also be happening to other clients in the same network as the monitor, as the same event that triggers RATE in the NTP server might trigger the firewall as well.

11

Probably not the firewall in this case, since i don’t apply ratelimiting at the FW.
Just wondering what the NTPd equivalent of the chrony ratelimit is:

chrony ratelimit interval 4 burst 8

12

What is the output of the ip -6 address show scope global | grep inet6 command on the offending server? I am wondering that the server is in the pool with multiple IP addresses, but the NTP server does not distinguish by the local IP interface for the rate limiting purpose, it sums multiple traffic from the same source to different destination IP.
(Alternatively, is there NAT6 configured on the router? That may have the same effect.)

13

@NTPman Will do that tonight.
Indeed it appears that rate limiting is applied on an Per-Source-IP basis. In case the same client (or monitor) contacts both server-IPs on the same physical server from the same source address this likely results in rate-limiting.
BTW: No NAT6 activated.
Obvious solutions are:

  • Deactivate ratelimiting (with increased exposure to abusive clients)
  • Have dedicated hardware for each server IP (leading to higher cost/e.g. increased maintenance effort).

Thanks for the help, im learning a lot!

15

Maybe a stupid idea, but why does the pool still send so many NTP-servers per DNS-request?
I mean, the servers that are give are all tested by many monitors, this wasn’t the case in the past.
Why no sending just 2 servers? (IPv4 only) and if needed 4 (ipv4+ipv6) and not large lists.

@ask , look how silly it is:

bas@workstation:~$ nslookup pool.ntp.org 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	pool.ntp.org
Address: 185.51.192.63
Name:	pool.ntp.org
Address: 45.87.76.3
Name:	pool.ntp.org
Address: 80.67.191.2
Name:	pool.ntp.org
Address: 162.159.200.123

bas@workstation:~$ nslookup 2.pool.ntp.org 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	2.pool.ntp.org
Address: 185.51.192.61
Name:	2.pool.ntp.org
Address: 80.67.191.2
Name:	2.pool.ntp.org
Address: 23.95.167.124
Name:	2.pool.ntp.org
Address: 178.215.228.24
Name:	2.pool.ntp.org
Address: 2a00:7580:60:211::52
Name:	2.pool.ntp.org
Address: 2a0e:f780:0:400::35
Name:	2.pool.ntp.org
Address: 2a00:7580:60:211::46
Name:	2.pool.ntp.org
Address: 2606:4700:f1::1

Sending this much IP’s is asking for trouble.

Why not change the ntp-clients to get just 2 IPv4 (and 2 IPv6) no more per request, BUT when the client can’t reach any, just let the client request the pool again on a different number. ‘backup’-pool if you like.

Say you give 2 lines per client:

1st: 1.pool.ntp.org (same as ntp.pool.org)
2nd: 2.pool.ntp.org

And the client only uses the 2nd if no response possible from the first.

That way IP’s can rotate faster without overloading some servers because 1mln devices get the same IP’s and spike.

We do not need to spread so many servers as the monitors check them all the time, unlike before.

Make the clients more intelligent and force them to check DNS for backup if needed.
Beware, that old/poor clients will get less IP’s and won’t hinder the network as much as they do now.

My 2cts. I mean, most people use only 2 DNS-servers and that works fine too. :grin:

16

That’s comparing apples and oranges. With DNS, a single answer is all that is needed. And unless it’s running into some kind of timeout because of too high latency or outright packet loss, the service itself is not dependent on the network characteristics, e.g., latency, jitter, path asymmetry.

With NTP, a “full” NTP client typically needs answers from multiple servers to maintain proper time. In addition to robustness to complete failure of a server or packet loss, NTP is sensitive to network characteristics, so multiple servers are needed to be able to filter out those network impacts (and server aspects) as much as possible, and come up with a best estimate of time.

17

Requested output is below:
~ $ ip -6 address show scope global | grep inet6
inet6 2a02:a46d:xxx/64 scope global
inet6 2a02:a46d:xxx/64 scope global
inet6 2a02:a46d:xxx/64 scope global

18

2a02:a46d:7ac:1:f690:eaff:fe00:be00 => in the pool (monitoring only)
2a02:a46d:7ac:1:f690:eaff:fe00:700e => in the pool (monitoring only)
2a02:a46d:7ac:1:213:95ff:fe0f:e507 => in the pool

So at the end, we found the explanation for the issue of hitting the rate limit.
A given monitor sends to all IP addresses the same quantity of packets. It does not matter if an IP address is set to “monitoring only”, it is monitored the same way. The NTP service receives all packets from the same source IP, and what it sees, is the triple quantity of packets relative to single IP configuration.

To fix the issue, just remove two IP addresses from the pool.

20

It this 3 machines or just 1?

Because if it’s just 1 and the NTP/Chrony is set to listen to all incoming, it gets everything 3 times.

21

It is only one system with three IPv6 addresses. Of course, it gets three times the traffic relative to having only one address in the pool.

22

Wonder if it’s the case if not understanding IPv6…