I’ve entered the world of NTP servers and recently spun up a server in Hong Kong which has just been added to the pool. I’m planning on starting low and slowly incrementing the Net speed as I don’t know if the CPU can handle 1 Gbit of throughput. Currently it’s at 12 Mbit.
I chose Hong Kong as I noticed there weren’t many servers there and read on the forums about how heavy Asia can be in terms of requests.
As I’m brand new to this, are there any tips that veteran NTP server owners would like to share? chrony config “must haves” or anything pertaining to NTP server ownership?
One thing I would suggest is to keep an eye on the traffic to and from the server. Have you got any monitoring in place? One thing you will see is LOTS and LOTS of requests coming from mostly one network. I am talking thousands of requests per second. To protect your server I suggest you look at chrony rate limiting and possibly iptables rate limiting. Sometimes the number of incoming requests can flood your link which stops the monitor checks and your server is dropped out of the pool for a while.
Have a look through the chrony.conf documents to learn about the rate settings. The settings I am using are:
# Allow requests from anywhere
allow
# increase client monitoring limit
# clientloglimit 300000000 uses about < 15%% of mem on 1GB instance
clientloglimit 100000000
# rate limit. Interval means 2 to the power of -3 = 0.125 so a packet every 1/8 second
ratelimit interval -3
One of the tools I use to monitor things is vnstat. It can produce a useful status image. My latest is
As the NTP protocol sends and receives similar sized packets the incoming and outgoing network traffic should be sililar but as you can see the light green lines on the right show that the incoming network traffic is far larger than the outgoing traffic. The rate limiters are busy at the moment.
My thoughts are to, as you are doing, increase the connection speed setting slowly and see how the server handles things.
Hi, I also just recently added 2 servers to the pool, one in the Asia region, other in the Netherlands.
The traffic patterns are wildly different between the two regions. The nl server handles 400-700 requests/second with a netspeed of 1Gbit, while the asia one handles 10k to 63k requests a second depending on the time of day with a netspeed setting of 500 Mbps. The data usage was 200 GiB on this day, with maximum of 44 Mbps of network speed in both directions. It’s great fun if you like handling lots of requests, but also pay attention so you don’t run out of your allocated bandwidth limits, as we know how the asia region is with that. The traffic peaks around 04:00 UTC and 12:00 UTC, while the lowest is at 21:00 UTC. Only around 1% is IPv6, while the netspeed for the IPv6 addresses are set to 3Gbit.
I just increased the netspeed to 1Gbit today. This will probably be the max I can handle while staying in bandwidth limits.
Also, turn off conntrack for the NTP ports. A reminder for nf_conntrack - #2 by lordgurke otherwise you will see messages like nf_conntrack: nf_conntrack: table full, dropping packetin your dmesg output.
If your firewall does NAT or tracks state in any way (as many Linux ones that allow established/related connections do by default), make sure to exclude NTP traffic from state tracking.
Rate limits are your friend. It won’t help you from the flood of packets, but at least you won’t be contributing to the flood of traffic in the other direction. With a rate limit in place you’ll see the asymmetry, and that’s the difference in the packets you’re dropping from abusive hosts/networks. In the US, I see it at about 20% with a limit of 1 packet/sec and a burst of 8.
Maybe there really are that many hosts behind a single large NAT IPv4 address, or maybe they are NTP packets that all have the spoofed source address of a dDoS target host to make it harder to trace the source of a dDoS. Either way, rate limits will help.
Unless you have heavy duty network gear and connection, add only IPv6 servers to the pool. There are far fewer cases of abuse and ill behaved clients. It’s night and day compared to IPv4.
Well it depends. On my own setup ancient Pentium3 computer and ordinary home fiber router (provided by ISP) can serve thousands of queries per second in tw IPv4 zone (my ISP does not provide IPv6 address to home users). The hardware cost is zero, however the utility bill is far more prominent… The setup takes near 60W in total, so running it all day long take me about 43.2 kWh per month, or about NT$180 (≒US$6) per month. YMMV.
I’m sorry, but most routers can not handle that as they do connection-tracking and you can’t turn it off.
I have an entire topic about this and how bad most routers are at this.
As I said in previous post, YMMV. What configuration worked in my gear may not still working in yours; I am just sharing a case report.
In Taiwan FTTH broadband usually served with PPPoE connection, which encapsulated multiple connections inside one data stream… Don’t know if this is available in your region, but it may worth a try.
Same here in Europe.
For fiber or dsl, all PPPoE connected.
Routers the ISP’s give you are poor and do connection tracking.
When you set speed high enough they reboot/slowdown/crash.
Unless you can disable tracking for ntp, it will happen sooner or later when using NAT.
and welcome to the pool.