Ntp server overload cause?

Hello,

I’ve run a chrony ntp server in the pool for about a year with no issues. The Net Speed setting has been on 10Mbits because I thought my residential cable network provider had data caps.

Turns out my ISP is unlimited (at least by their marketing department) so I tried turning up the Net Speed setting to 1000Mbits.

It seemed fine and the requests per second on port 123 ramped up, as expected.

But the normal web browser use on the Ubuntu Linux machine started becoming unresponsive, as did another nearby windows machine.

It seems the increased NTP packets interferes with DNS. Web pages are slow initially but once they start they flood in. Bandwidth is not the issue. It’s DNS.

The problem becomes so bad it starts reporting poor scores on the pool. To the point where it drops out. Bizarre?!? The machine is not overloaded. Cpu load and even network bandwidth is fine.

It’s mainly DNS. Outbound nslookups fail during these scenarios.

Ubuntu 20.04 Linux is fully updated, and mostly stock except for some standard internet hardening steps to ensure security. Chrony is configure well as a stratum 1 server using GPS time.

What’s the deal here? Why is high NTP traffic on port 123 interfering with DNS traffic?

Marc

Hi Marc,
sounds like your routers UDP connection tracking table fills up. Most plastic home routers have only limited ressources for tracking connections and I guess the UDP table fills up with all the incoming/forwarded UDP 123 connections, and now DNS which uses UDP 53 by default gets affected.

Regards, Gunnar

4 Likes

Gunnar,

You might be right on this. I was skeptical at first because network pipeline bottlenecks have multiple locations along the way. My ethernet card has a low default ring buffer size. But if 2 machines have a network bottleneck, then it might be the router. I prepared a few monitoring tools and tried again while monitoring network traffic on Ubuntu with:

  • netdata
  • ifconfig
  • ethtool
  • netstat
  • dropwatch

None of them indicated any dropped or truncated packets in the desktop linux machine. The udp packets per second went to 400-600 at the Ubuntu machine but there’s no telling how many were dropped or not passed along because of the router.

Almost immediately DNS was hindered and the router CPU temperature went up and it’s load indicated near 80% on the bar chart. The 3 load numbers also indicated the router was nearing capacity. Normally the router load is quite low, like 15% on the bar chart.

It’s a curious situation because, yes, it is a plastic box, but it’s a Linksys 1900AC with DD-WRT installed from the factory. Linux on routers do a pretty good job.

My next investigation will be how to tweak dd-wrt to improve bandwidth. It’s a good router and there should be dials and knobs to increase buffer sizes for improved performance.

I think it’s the router and am surprised.

1 Like

@gunnar, your comments were spot-on.

My Linksys AC1900 router has 256MB ram so there’s plenty of memory, but still, there are 600-1000 udp messages coming in per second for Net Speed setting at 100Mbits, so the max network connections table fills quickly.

My max connections are around 32,000 and I dropped the UDP timeout to 30 seconds. This makes the router handle the traffic but it was holding a giant list of 10,000 or 11,000 active IP connections the last I looked.

There’s no conflict between DNS on port 53 and NTP on port 123. It works but it’s just slow.

So I’ve dropped my Net Speed setting to 10Mbits and it seems to work just fine.

Thanks,
Marc

1 Like

Hi Marc,
so you are running DD-WRT (great choice), so may have found this article: Router Slowdown - DD-WRT Wiki
They show how to increase the internal tables in the kernel to allow more IP/Port mappings. Decreasing the timeout may also help to clear the table faster.
But as you found out ultimately you will run into another limit and this time seemed to be the CPU in the router because NAT doesn’t seem to be hardware accelerated. So now other processes like DNSmasq are slowing down :frowning: Unfortunately sometimes you lose, sometimes the other team wins…

Regards, Gunnar

1 Like

Instead of messing with timeouts and memory limits, it would be better to disable tracking for the NTP packets completely. I’m not familar with DD-WRT. On OpenWrt custom rules can be specified in /etc/firewall.user as iptables commands, e.g.

iptables -t raw -I PREROUTING -p udp -i eth0 --dport 123 -j CT --notrack
iptables -t raw -I PREROUTING -p udp -i eth1 --sport 123 -j CT --notrack
2 Likes

Hello @mlichvar , this is an ideal solution. I’ve seen DD-WRT customizations with iptables, so I’ll have to investigate. This is the best solution because tracking is not important at all for so many ntp requests per second. It could offload this from the CPU altogether.

I’ll investigate and test.

Well, apparently disabling connection tracking disables NAT, which makes sense, but it also may disable port forwarding.

I’ve tried both command versions to disable connection tracking as follows:

iptables -t raw -I PREROUTING -p udp --dport 123 -j NOTRACK

and

iptables -t raw -I PREROUTING -p udp --dport 123 -j CT --notrack

Both of these commands are accepted by DD-WRT without complaint.

Both stop connection tracking. Great.

Except wait.

Both also stop traffic NTP from being forwarded through the DD-WRT router to the NTP server.

No more NTP traffic. (?)

Question: Is is possible to disable connection tracking and still allow port forwarding?

and I’m not the first to discover this issue, but these entries are difficult to locate, for example with this subject: :joy: :joy: :joy: