hi. i need the list of monitors trackers to add they into whitelist. cuz after DDoS some addresses was added into the blacklist and now my score down to -90
How are pool clients supposed to get time from a server which responds only to monitors?
4 Likes
that’s not the main question today.
It’s the right answer though. Or maybe you aren’t asking the right question.
The monitoring system is designed so the monitors send NTP queries in a pattern similarly to a reasonably well behaved client, so if they get rate limited that’s a configuration problem on the NTP server.
If your server is blocking because of the fallout from a one-off event, could you just reset the blocked addresses?
3 Likes
Hi and welcome to the pool.
May I ask where in the world your server was running? The traffic patterns across the world are significantly different so that some locations are very difficult to serve so your server gets flooded as you have seen.
A few months ago I setup a server in Manila. It lasted a few days, if that. Below is the eth traffic from that server.
The first down spike of traffic was with no rate limit at all. The number of requests was more than the amount of traffic a chrony instance can handle (about 30 ~ 40 Mb/sec from my experience) the monitors could not get a response from the server so removed it from the pool. They did their job correctly.
I then added some rate limiting - Now people have different thoughts about that so feel free to start a new thread and I am happy to discuss. The rate limiting was 8 requests per second from an ip address and port combination - srcIp:srcPort. I then again set the connection speed to 512K. The second spike in requests flooded the whole instance. From the look of things there was some networking limit of 100Mb/sec. The incoming requests flooded the link, monitors could not get a response so correctly removed the server from the pool. It has not been put back in the pool as I, so far, have not come up with a way to add it without the server being flooded. Happy to hear ideas from people.
What I saw - There were a number of ip addresses that were sending up to/over 5000 requests per second all from the same ip and source port. People talk about CGNAT - There could have been a LOT of clients behind CGNAT but at 5000 per second how is the NAT going to send the requests back to the correct client. Interested to know.
It could have been a DDOS as some of the ip addresses were Starlink - no idea why people would want to disrupt that service.
What ever it was it meant that having an NTP server(VPS) in Manila was not going to work.
If I had setup a whitelist to avoid the flood as you seem to want to do then two things:
- Who would be on the whitelist? The monitors and who else. A client in Manila makes a request to the pool and asks the server provided by the pool dns system but they are blocked. The pool fails.
- In this location, even if there was a whitelist, if the servers ip address was given out then incoming requests would likely flood the server anyway to the point the monitors would remove it anyway.
Anyway happy to chat and see if we can get your server running in the pool.
1 Like
Hi Gombadi, the Philippines are pretty underserved indeed. I think the number of servers right now is in a spot where the net speed setting doesn’t work very well.
Yes agreed. Looking back at the notes I took it was actually 128Kb connection speed. I guess that any server added to the pool, as soon as it is known gets hit by a number of large clients and their requests. Real requests or DDOS does not really matter as it still floods the system.
Interested to know if anyone has any thoughts on how to get a server up and running here.
ask, @ gombadi I can answer both
when I add monitors to the white list - I will be able to keep my server status within the required 10 points, which means my server will be available to good people as a time server.
Now that my server will always be available to good people, all I have left to do is to fight off DDoS attacks and evil people. And this task is much easier than the first one, because I will simply block addresses with evil people for some time until they lose their appetite.
I hope that they will give me a list of monitor address pools and I will again include my server in the pool of time servers for good people.
over the last two days I have beaten off about 30,000 addresses from which the attack was carried out, almost all of them turned out to be from the servers https://www.datacamp.co.uk/.
Meanwhile, I noticed that only one pool monitor was not blocked and shows my status above 10 points.
I am putting the list of blocked addresses here for now GitHub - cybermerlin/blacklist: Black list. Scammers and attackers.
What is the criteria for adding an IP address to your blacklist?
Your GitHub page says “this is a file with SSH knockers and some ports scanners + http DDoS attackers and vulnerabilities scanners”. I’m fairly certain that the pool monitor servers do not do any of the above, so there should not be a reason to whitelist the pool monitor servers.
As others have said, if you need to whitelist the pool monitor servers you are doing something wrong.
1 Like
spam request
ssh knoking
scan all all massive range ports
sending through http with method not in allowed list (for example base64 encoded strings)
sending POST data to undefined pages
seeking undefined pages
sending sqli encoded data or try to use another attacks
too many identical requests per 60 second from 1 address
The list of things that make a remote ip address a “bad person” does not seem to include ntp requests. That suggests you are saying if an ip address send an invalid http request you will tag them as bad and block ntp requests from that address.
With CGNAT there can be thousands of users all sharing the same source ip address. One or more could send an invalid request therefore all of them are going to be blocked. Does that mean that your server is therefore a “bad server” from the point of view of the pool clients? You are advertising your ip address as part of the pool but blocking requests because of something that someone else may have done.
Interested to know how many of those addresses were making ntp requests?
1 Like
Indeed. As well as what constitutes “too many” and “identical requests”, respectively.
Apart from the already mentioned issue with CGNATs, there are known misbehaving (malfunction, not necessarily malfeasance) types of clients. See, e.g., this thread, and the tool to detect some of those misbehaving clients referenced therein.
As also mentioned before, blocking NTP requests for alleged infractions related to non-NTP protocols, or worse, for alleged infractions by other clients sharing the same IP address for purely technical reasons (CGNAT), seems very much like throwing out the baby with the bathwater. And whitelisting the NTP pool monitors doesn’t really solve the underlying issues with such a heavy-handed, indiscriminate approach.
Case in point, the current monitors aren’t even on the supposed blacklist as of right now, pointing to potential further issues unrelated to the monitors that would warrant looking into, and understanding first.
1 Like
I’d like pcap samples of the abusive traffic. In additional to the Fortigate clients, I’ve seen problems from L2/L3 network loops, timesyncd bugs, and a few other strange ones.
There is no unique definition of what constitutes “abusive” for NTP. NIST mentions “more frequently than once every 4 seconds”. There is also iburst which is near that rate. I’ve focused on clients that send hundreds or thousands of requests per second.
For this discussion forum it would help if the IPs you’ve identified as NTP abusers were in a separate list.
2 Likes
you are right only in one thing - there can be several hosts behind one address.
However, you do not take into account that if whoisIP reveals that the address belongs to the hoster, and not the provider, then this address is the corporate infrastructure. But in any case, the owner of the address is responsible for the purity of its address. This means that this owner allows his infrastructure to be used for evil intentions or even provides it for these purposes.
This was the first fact.
Secondly, if attempts are made from the address to scan, hack (for example, brute force ssh) or something like that, this does not mean that it is worth forgiving it and providing it with a service that will facilitate the hacking of my or another host.
Thirdly, using purely NTP, I rejected addresses that sent 10 or more identical packets from one address.
Identical - means the datagram is the same, the data is the same. Under no circumstances will it happen that different hosts behind one NAT will send exactly the same requests - it is either one host or a botnet.
I could write a lot of things here and give clear instructions, but they will stop working in about a few minutes after evil people read them. Therefore, my script adapts to intervals, sequence and structure. But there are absolutely clear basic schemes - for example, if you are scanned by ports - this is an evil person, or if they knock on your ssh or send a syn packet, and then break the connection. And also if the packet size is made large, like in ping or ntp.
You can tolerate these attacks, but I prefer to turn off several thousand servers for some time to allow good people to use the services that I am ready to provide.
In our society, until all people think about improving the future - there will be not only good people, but also bad people. I do not want to punish bad people by redirecting their own traffic to them yet, so I will simply turn them off temporarily until they lose their appetite.
But if the bad people become even more embittered, then I will simply redirect their incoming traffic to them using the “one-to-many” scheme, clogging the channel, using a torrent network and VPN.
1 Like
I recently caught 5000 requests from one address to my NTP server in 2 seconds - this is clearly an attempt to crash my server. I don’t know why, but there are evil people who don’t want our community of free NTP servers to exist. And I notice such attacks from large cloud infrastructure providers.
I am generally surprised why I have done everything to reduce the likelihood of my gateway being used by my users with malicious intent, while others do not do this and allow such actions
1 Like
I guess it starts with different people having perhaps more differentiated views about this, and more differentiated approaches to deal with any issues considered by them to be issues.
Each to their own, but your approach apparently impacting how your server actually serves clients, as reflected by the monitors apparently being unhappy, seems to indicate that a more differentiating approach might be in order.
Regarding the 5000 requests you “caught”, have you perhaps caught them in the form of a traffic capture that you could share, as @stevesommars asked?
The thread I pointed to, and the tool it references, and the background of @stevesommars asking about the traffic capture all have the same point, that some clients that seem malicious simply are malfunctioning, without any malice intended. That is why it would be interesting to get a traffic capture of the offending traffic that you try to fend of.
E.g., for some implementation, it is not uncommon to see packet rates of several thousand packets per second. E.g., worst I could find offhand, from the tool mentioned above:
2025-02-12T15:04:06Z 88.200.208.86 port=123 rate=31487 client=fortigate
2025-02-12T15:04:07Z 88.200.208.86 port=123 rate=32064 client=fortigate
2025-02-12T15:04:08Z 88.200.208.86 port=123 rate=31456 client=fortigate
2025-02-12T15:04:09Z 88.200.208.86 port=123 rate=32109 client=fortigate
2025-02-12T15:04:10Z 88.200.208.86 port=123 rate=31312 client=fortigate
2025-02-12T15:04:11Z 88.200.208.86 port=123 rate=32398 client=fortigate
2025-02-12T15:04:12Z 88.200.208.86 port=123 rate=32030 client=fortigate
2025-02-12T15:04:14Z 88.200.208.86 port=123 rate=32155 client=fortigate
1 Like
You mean floods like this
# time tcpdump -tnl -i eth1 -c 10000 -Qin udp port 123 | awk '{print $2}' | sort | uniq -c | sort -rn | head -n 30
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10000 packets captured
15757 packets received by filter
0 packets dropped by kernel
5260 180.195.198.228.3118
239 120.28.197.196.40484
...
74 112.203.133.19.26786
real 0m0.225s
user 0m0.066s
sys 0m0.102s
That is in a capture of 10,000 packets over 0.2 seconds, 5260 packets were from the same source ip and port. Can you show us your packet captures as interested to see what packets you are receiving.
What I have on some of my servers
iptables -A rate_it -p udp --dport 123 -m hashlimit --hashlimit-name NTPRATE --hashlimit-mode ${HASH_MODE} --hashlimit-srcmask 32 --hashlimit-above ${HTABLE_RATE}/sec --hashlimit-burst ${HTABLE_BURST} --hashlimit-htable-expire ${HTABLE_EXPIRE} --hashlimit-htable-size ${HTABLE_SIZE} --hashlimit-htable-max ${HTABLE_MAX} -j DROP
If one srcIp:srcPort sends more than 8 requests in one second it is blocked for a max of 3 seconds. What does it do
Chain rate_it (1 references)
pkts bytes target prot opt in out source destination
3777K 287M DROP 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:123 limit: above 8/sec burst 10 mode srcip-srcport htable-size 32768 htable-max 262143 htable-expire 3000
The counter is reset every hour. Above is about 1 hour.
You mention that the bad people will loose interest after a while. My NTP servers have been hit for months.
Are the monitors being blocked on my server - No as they send less than 8 per second. Do I need the ip addresses of the monitors so I can whitelist them - again no.
Interested to know the details of what packets you are receiving and the defences you have in place.
And sorry I still do not see that whitelisting the monitors is the solution that you need.
3 Likes
you are right, you can do without knowing the addresses of the monitors. Let the attackers block all my traffic, and let my users waste their nerves and time, because hundreds of thousands of requests come from the pool without control and everyone around suffers as a result.
Okay. Until the traffic from the pool is balanced normally and is not checked for evil people, I will not waste time to ensure that the monitors do not go into the negative.
If some addresses ended up on this blacklist, then everything is bad there. God knows I tried, but I have already spent two days trying to find a solution that would suit everyone.
I did not collect traffic to share it, because this would load the gateway and provide a point of failure for attackers.
1 Like
NTP requests sometimes exceed the threshold that individuals consider abusive. There is no consensus on what that threshold rate is. There is also no consensus on the throttling mechanism: total block, rate-limit, …
I’ve worked with a few folks in the NTP Pool to investigate high rate NTP requests. Another example is the “1N14” clients. Most of the “abusive” sources seem to be bugs, or lazy/inconsiderate clients. One such example is simultaneously restarting hundreds / thousands of VMs but pointing the time requests traffic to a single NTP pool server. [The admins should configure a local stratum 2+ server instead.] But these probably not intended to crash the NTP server.
The most prominent malicious attack that I am aware of is the monlist vulnerability in the reference NTP distribution. This was fixed years ago. I’d also classify NTP reflections as a potential problem.
This is not a simple discussion. Being based on unauthenticated UDP, NTP is inherently vulnerable to abuse. Some operators favor aggressive rate limiting. Some people see NTPsec as the answer. I don’t have a good solution, so I study the abusive traffic (tcpdumps) and lobby with the responsible party for remedies. Sadly, it is often impossible to identify or communicate with the responsible party.
1 Like