I’ve run a chrony ntp server in the pool for about a year with no issues. The Net Speed setting has been on 10Mbits because I thought my residential cable network provider had data caps.
Turns out my ISP is unlimited (at least by their marketing department) so I tried turning up the Net Speed setting to 1000Mbits.
It seemed fine and the requests per second on port 123 ramped up, as expected.
But the normal web browser use on the Ubuntu Linux machine started becoming unresponsive, as did another nearby windows machine.
It seems the increased NTP packets interferes with DNS. Web pages are slow initially but once they start they flood in. Bandwidth is not the issue. It’s DNS.
The problem becomes so bad it starts reporting poor scores on the pool. To the point where it drops out. Bizarre?!? The machine is not overloaded. Cpu load and even network bandwidth is fine.
It’s mainly DNS. Outbound nslookups fail during these scenarios.
Ubuntu 20.04 Linux is fully updated, and mostly stock except for some standard internet hardening steps to ensure security. Chrony is configure well as a stratum 1 server using GPS time.
What’s the deal here? Why is high NTP traffic on port 123 interfering with DNS traffic?
Hi Marc,
sounds like your routers UDP connection tracking table fills up. Most plastic home routers have only limited ressources for tracking connections and I guess the UDP table fills up with all the incoming/forwarded UDP 123 connections, and now DNS which uses UDP 53 by default gets affected.
You might be right on this. I was skeptical at first because network pipeline bottlenecks have multiple locations along the way. My ethernet card has a low default ring buffer size. But if 2 machines have a network bottleneck, then it might be the router. I prepared a few monitoring tools and tried again while monitoring network traffic on Ubuntu with:
netdata
ifconfig
ethtool
netstat
dropwatch
None of them indicated any dropped or truncated packets in the desktop linux machine. The udp packets per second went to 400-600 at the Ubuntu machine but there’s no telling how many were dropped or not passed along because of the router.
Almost immediately DNS was hindered and the router CPU temperature went up and it’s load indicated near 80% on the bar chart. The 3 load numbers also indicated the router was nearing capacity. Normally the router load is quite low, like 15% on the bar chart.
It’s a curious situation because, yes, it is a plastic box, but it’s a Linksys 1900AC with DD-WRT installed from the factory. Linux on routers do a pretty good job.
My next investigation will be how to tweak dd-wrt to improve bandwidth. It’s a good router and there should be dials and knobs to increase buffer sizes for improved performance.
My Linksys AC1900 router has 256MB ram so there’s plenty of memory, but still, there are 600-1000 udp messages coming in per second for Net Speed setting at 100Mbits, so the max network connections table fills quickly.
My max connections are around 32,000 and I dropped the UDP timeout to 30 seconds. This makes the router handle the traffic but it was holding a giant list of 10,000 or 11,000 active IP connections the last I looked.
There’s no conflict between DNS on port 53 and NTP on port 123. It works but it’s just slow.
So I’ve dropped my Net Speed setting to 10Mbits and it seems to work just fine.
Hi Marc,
so you are running DD-WRT (great choice), so may have found this article: Router Slowdown - DD-WRT Wiki
They show how to increase the internal tables in the kernel to allow more IP/Port mappings. Decreasing the timeout may also help to clear the table faster.
But as you found out ultimately you will run into another limit and this time seemed to be the CPU in the router because NAT doesn’t seem to be hardware accelerated. So now other processes like DNSmasq are slowing down Unfortunately sometimes you lose, sometimes the other team wins…
Instead of messing with timeouts and memory limits, it would be better to disable tracking for the NTP packets completely. I’m not familar with DD-WRT. On OpenWrt custom rules can be specified in /etc/firewall.user as iptables commands, e.g.
Hello @mlichvar , this is an ideal solution. I’ve seen DD-WRT customizations with iptables, so I’ll have to investigate. This is the best solution because tracking is not important at all for so many ntp requests per second. It could offload this from the CPU altogether.
Yea, thats right.
Also most residental connection don’t have a static IP (
Your computer must have a static IP address and a permanent internet connection. It’s very imporant that your IP address doesn’t change or only does so extremely infrequently (say once a year or less)
) and / or have a symetric connection. If you have an asymetric connection this could also hassle if to much upload happend…
bah to both of these posts. My home cable connection works great and serves time to thousands of requests per day. my ip address hasn’t changed in a over year.
there’s no major issue with adding a home router to the pool. the limit I’ve discovered is a cap on max number of ntp requests per second. it’s an issue related to iptables requirement for connecting tracking with nat. nftables has a way to selectively eliminate tracking for port 123 but I’m waiting for the new openwrt version 21 to stabilize before pursuing.
for now it works great and I keep the traffic setting at 10Mbits/s on the pool server settings. I get between 80 and 200 NTP requests per second and it works great.
Unfortunately sometimes you lose, sometimes the other team wins…