Hello everybody,
I use ip’s block lists, my question is if it is possible to know the ip’s of the ntp monitors? Knowing the ip’s I can create a suppression list and eventually they won’t be blocked, so they won’t condition the score of the ntp server. Thanks.
The monitors query the NTP servers fairly infrequently. If your NTP server blocks such monitor nodes, your NTP server is misconfigured. Whitelisting monitor nodes is not a good idea.
Hy avij, Thanks for the answer.
Do you really need block lists at all? It’s quite rare that badly behaved NTP clients actually change their behaviour when they stop receiving replies, and it’s unlikely that it would have significant impact on your server(s) unless you are serving a huge number of clients.
If you intend to use a server in the pool, you should not use the “whitelist” feature to allow only certain IP addresses. By doing this, your server would be considered “healthy” by the pool as it would be responding to monitoring stations, but to real users it would be unreachable. This in practice means circumventing the pool monitoring mechanism.
Correct we would not give you the monitoring station IP’s for this reason. There is no reason for you to know what is a client and what is a monitor. If you have a server in the pool it should answer the same for both the clients and the monitor. If you are using whitelist then you are not providing a service to the pool and have no use to everyone. Your server needs to be open to the world if you have it in the pool.
Thanks!
Hi everybody,
thank you all for the explanation.
As a server operator, i do rate limit all clients by default. I rate limit those clients not using the NTP port more aggressively, since it’s more likely that user ports are used in case of abuse. Then, depending on the specific case, knowing the addresses of the monitors can be helpful. Only the server operator can decide what is acceptable traffic.
I do ratelimit also, if they hit harder then 2^2 seconds (bursting above 6 times will not be answered) all the time they are blocked.
But there is no IP blocking, for nobody.
You should not do such when being a part of the pool, as when you do, people will get your server via the DNS and you block them, not quite the idea of the system.
In Chrony this is a simple task: ratelimit interval 2 burst 6
Abuse will punish itself if they try to go faster.
Ps. my ratelimit is this low as I have 2 stratum 1 servers that poll faster to keep them as accurate as possible so they can compare PPS signals. Other severs are limited to 2^4
Ps2. you can also set a bitrate at the manage-servers page, just set it lower if you find the load too much.
I still use the following strategy. There are always a few IPs on the blocklist as a result - at the moment of this post, there are currently four. That’s on the lower end of what is typical for my server.
In my opinion it’s worthwhile having some sort of automated throttling of clients, even if just as a defense against bugs. As an example, 20,000 packets / sec for each problematic client is not insignificant - see the Fortigate bug below. Rules that chop the peaks off that kind of load can be quite helpful.
Note. There are at least two philosophies for dealing with huge bursts. ntpd from Network Time Foundation can strictly limit the response rate sent to a client. Miroslav pointed out that such rate limiting opened a denial of service channel. Thus chrony’s model for dealing with huge bursts is different, see the ratelimit directive of chrony.conf.
This is the case where I think IP blocking potentially makes sense—when you can drop the traffic further upstream.
For a while I would periodically review NTP’s monlist and manually add iptables rules to drop egregiously broken clients, but I realized that (maybe outside a trip through userspace?) I wasn’t really saving anything by letting iptables drop the packets on the server instead of ntpd, but I was driving myself crazy trying to maintain a bunch of rules. In the case where you can block clear abuse further upstream, that makes more sense than what I was doing.
There is weirdly much junk that gets dropped by simple ratelimiting, though:
chronyc> serverstats
NTP packets received : 2536035832
NTP packets dropped : 589590870
This is with ratelimit interval 2 burst 8 leak 3
in place which I consider to be fairly permissive. It’s not really problematic, but has always struck me as odd. How are there so many bad clients sending so much traffic?
So many bad clients? See my article that describes misbehaving FortiGate(Fortinet Firewalls) acting as NTP clients. Over three years later we still see a small number of these sending NTP bursts.
Each client sends a short burst of 500+ NTP requests / second for a short interval. Worse, on occasion 10-60 clients burst simultaneously.
Worse, the software bug apparently affected another class of Fortinet hardware, some sort of VPN client. Today I see about 80% of the total IPv4 NTP traffic for lt.pool.ntp.org (Lithuania) coming from these Fortinet devices. The 80% number has been fairly constant for at least the past 1.5 years.
The problem is not limited to lt.pool.ntp.org. About 2% of the total NTP traffic for a server located in London comes from misbehaving Lithuania clients. I expect that other NTP Pool servers in Europe are also affected.
Fortinet support no longer responds to my emails.
There are other offenders, Amazon, Cellular carriers and the like. Plus there are occasional network loops and software bugs.
And in IPv4, there’s a sliding scale of how abusive individual clients are vs. how many clients are using NAT behind one IP address. If someone boots 100 VMs at the same time, you might receive a large burst of packets and then a query on average every 0.64 seconds.
No standard client of the pool will be blocked, only abusive clients, which are legion. But i white list the monitors, firstly because they are trustworthy, secondly because they need unrestricted access to keep my servers in the DNS pool. Again, only the server operator can decide what is acceptable traffic.
You shouldn’t be whitelisting for the monitors, nor is there any need for you to do so. If your normal traffic management strategy restricts things to the point it would result in the monitors kicking you out, then your server is by definition unsuitable for inclusion in the pool, as it is providing a very low quality service to clients.
The whole point of the monitoring system is to ensure that systems which are unable or unwilling to provide a sufficiently robust service are not advertised to clients.
Exactly. I also have a very thorough IP blocklist in place, but I have configured my firewall to ignore the blocklist specifically for NTP traffic. My NTP server exists on a separate subnet with tight controls on what it can access in my network, if you’re concerned about security I’d recommend you do this too.
I also have the same situation as you. There are many ways to kill flies.
The transparency we should all have must be reciprocal.
My ntp server ip is public, why aren’t the monitors ip?
For Security reasons ?
An administrator has no problem blocking any malicious ip including blocking the asn ip.
When a server, which we all know many are not exclusive time servers, is attacked, the administrator is not concerned with the quality of the service since it is being harmed by malicious IP’s.
I think it would be beneficial to know at least the asn lookup of the ip’s monitors.
If a monitor briefly gets caught up in a temporary rate limit it will handle that. As @erayd and others said, if your abuse prevention blocks the monitors you are blocking too broadly to be useful as a public server in the pool system.