Cool, problem solved then, this topic can be closed.
Seriously though, solving this issue is primarily in Russian hands. Don’t expect too much help from “the pool” or the rest of the world. Given the current political climate, I would think that both Russians and non-Russians would prefer that Russians take care of this problem by themselves instead of asking for help from abroad. What you in .ru can do right now:
Ask around on Russian tech forums and media outlets etc. and see if large ISPs or large universities or large other public organizations would be willing to set up a couple heavy duty NTP servers. These organizations should not be afraid to receive inbound traffic measured in hundreds of Mbits/s. This might help in making the traffic levels to smaller player NTP servers tolerable again.
This applies to all NTP pool server operators in Russia: Make sure your own server is configured properly. Most importantly, check that your server, firewall or router does not do connection tracking for UDP/123 traffic. It is possible that with the previous traffic levels the servers did OK even with connection tracking in place, but with the increased traffic levels the connection tracking became a problem. See “conntrack -L” or “cat /proc/net/nf_conntrack”. See also “dmesg” if there are messages about connection table overflowing. If you see tracked connections, something like “iptables -t raw -A PREROUTING -p udp --dport 123 -j CT --notrack” and “iptables -t raw -A OUTPUT -p udp --sport 123 -j CT --notrack” should do the trick. Most servers should be able to handle around 50k qps without particular problems, if configured properly. Smaller embedded devices (Raspberry etc) and routers may have lower limits, though. This of course assumes the server has sufficient outbound bandwidth, eg. that 50k qps rate translates to around 50 Mbit/s.
If there are abusive clients, either intentionally or not, report them to the IP address owner’s abuse address (check whois). In addition to the media sexy DDoS claims, it is also very possible that some buggy application, even a phone app, creates tons of unnecessary NTP requests. It would be advised to record some traffic with tcpdump for evidence and further analysis, like:
tcpdump -c 100000 -w ntp.pcap inbound and ip and udp and dst port 123
tcpdump -nn -r ntp.pcap | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head
tcpdump -nn -r ntp.pcap host 1.2.3.4
Then there’s also the possibility that the number of abusive clients hasn’t really increased, but the increase in traffic was simply caused by some major NTP server operator pulling the plug, as suggested earlier in this and other topics.
(edit much later as an addendum to point 2): With this configuration your iptables won’t do connection tracking:
Ah, I apologize for any misunderstanding. What I meant was that this decrease in monitoring scores was not due to network blocks caused by firewalls (at least not the decisive factor), but possibly due to the server indeed experiencing overload interruptions under certain circumstances.
I can no longer add my VPS to the pool. Even setting the minimum bandwidth (512kbps) results in traffic exceeding the agreed bandwidth by more than 400 times and requests exceeding 800k pps within an hour—followed by being overwhelmed by DDoS cleansing alerts.
Every day, I receive tons of messages repeatedly: traffic greatly exceeds the warning threshold, triggering the service provider to cleanse—traffic drops to normal, alert lifted—once again exceeding the threshold and triggering cleansing. This not only makes my blog completely inaccessible but also results in receiving over thirty alert messages within a day.
However, I am curious if NTP can accept dynamic IPv6 addresses bound to domain names? Due to the severe shortage of allocated IPv4 addresses in China, home broadband almost never gets a dedicated IPv4 address—generally using NAT technology, where one or more IP addresses are used for an entire residential community. But IPv6 resources are abundant. Every broadband connection can be assigned a dedicated IPv6 address! And home broadband resources are relatively plentiful (usually 500Mbps downstream/50Mbps upstream). However, due to broadband restrictions, it is necessary to rebroadcast every 3-5 days to obtain a completely new IP address. But I have a domain name and DDNS service. Can I bind the dynamic IP to the domain name and then add the domain to the pool?
No, that won’t work. Adding a server is done by IP address only, and it’s unlikely to change as ntp clients often hold on to an address as long as the ntpd process is running.
Chrony already handles unresponsive servers better than ntpd, and there are improvements coming to the reference implementation ntpd, but it will take a while before all the clients are upgraded.
What is your VPS’ configuration? I am thinking about buying a bit more expensive one as 1vCPU, 1G RAM bails out out of pool in several minutes, look at the chart above.
The smaller one bails out of pool when its’ score exceeds 10. During night the second one should be approved by monitors and it should be clear if it can handle loads, maybe cheaper/more expensive one is needed.
I am going to write a one-liner which will init a Pool node on naked Debian system.
As a reminder, chrony and ntpd are single-threaded, so multiple cores won’t help that much. This would be an excellent time to have a look at rsntp, which acts as a multithreaded frontend for an actual ntp server. I have not tried rsntp myself yet, but its description seems interesting for this use case.
Thanks, I’ll try tomorrow.
Now I have 2 VPS which only task is seeding time to the Pool. One of them costs 6 rub. (~$0.06) daily, the second one - 2.5rub. (~$0.025) hourly. I’ve posted an article about situation to big Russian IT portal, if it will be approved by moderators, the problem should be solved quickly.
Ok, no problem. I thought that the guy above said that the thread is politically-motivated and is not interesting to anyone else, and the second guy said that we should help ourselves on our own. As it is not interesting - we could speak native language with each other not wasting time on translating (though all IT specialists in Russia have fluent English, at least reading/writing).
I do not agree with the statement, I think this thread is not politically motivated. I think it is pure technical. While everybody could translate Russian to English with some translator, I would appreciate if we stick to English. (I am neither native English speaking, by the way.)
