I ran into new/old router problems.
Don’t worry my router runs fine, but I noticed some hickups from time to time.
Turns out to be buffer-bloating.
You can test yourself:
Often buffers are set far too high, same with OpenWRT, they’re TX-buffer is set at 1000 (no idea what it means, but it’s too high).
It’s not bits or bytes.
But after a lot of testing, I set the TX-buffers to 100 on all interfaces and behold, the bloating is gone.
As I noticed hickups in my network under load.
What happens is an application, could be NTP, get’s a lot of requests and the buffer is too big.
The application keeps accepting requests but the buffer can’t be emptied so it keeps full and delays traffic, that starts to stutter or worse.
Because the buffer keeps accepting but can’t deliver things go wrong.
Many routers have set the buffers too high for no reason.
Test yourself, as it will overload the buffers and show the problem.
It helped me to tune my OpenWRT better as it’s wrong on the buffers.
This is an old, known problem. Qdiscs like fq_codel and cake tend to keep buffers small on interface level and are supposed to battle this problem. I currently run fq_codel
With an asymmetric link speed like 100/30 Mbps it would help if you limited the speed on your router to the NTP server to about 25 Mbps or so, dropping NTP packets above that rate, so spikes in NTP traffic wouldn’t clog your slower uplink. That would be policing, not shaping, easier on the CPU and delaying NTP packets in one direction would corrupt clients’ NTP measurements anyway.
It is limited to 512Kbps for my private server, else I get far too much requests.
I only did set it higher for testing purposes.
In Belgium setting it much higher will result in massive requests because of too low number of servers.
Apparently I need to highlight that if your server gets around peak 3500 queries per MINUTE at a 1.5Mbit setting, that is absolutely nothing that can be considered “massive”. 3500 queries per minute translates to around 58 queries per second. That is peanuts. Any properly configured system should be able to handle that amount of queries easily. If your system can’t handle that query rate, set your net speed to the lower value.
You’re missing the point, it’s the spikes that happen often, where it goes up from the normal rate of about 60/sec to >800/s.
We all see it because we have to low number of servers in Belgium.
If you paid attention, you have seen the graphs I posted about it.
My system can handle a far bigger load, but it’s the spikes that makes my home-network unusable.
Read the other topic about routers unable to handle the load.
I found that moving to OpenWRT and using the code of @mlichvar makes that it works.
It’s commercial routers that run into troubles, even over such a small load.
I tested the latest FritzOS for my 5690Pro, to see if they finally fixed it, it rebooted 2x within 24 hours over NTP-spikes and gave me DNS-resolve errors almost every hour.
They call that modem/router PRO! and should be able to handle fiber. Yeah right, it’s rubbish.
Like many others. That is the other problem, the spikes.
See the big spike…that causes problems on commercial routers.
Even the peaks are using less than 3% of your upstream bandwidth. This shouldn’t be a problem.
As I understand it, you may get the traffic peaks when your server is included in the main be.pool.ntp.org DNS zone, which will happen from time to time. The 0, 1, 2, 3 subzones probably attract less traffic.
There are 9 DNS servers that serve pool.ntp.org zones and each response has the IP addresses of 4 NTP servers. This means that at any specific time there can be at most 36 different NTP servers included in the be.pool.ntp.org responses. Here’s the distribution of those IP addresses at this very moment:
$ for i in a b c d e f g h i; do dig @$i.ntpns.org +short be.pool.ntp.org; done | sort | uniq -c | sort -rn
6 162.159.200.123
4 45.87.76.3
3 94.142.246.192
3 45.87.78.35
3 45.138.55.60
2 85.163.168.227
2 80.200.247.170
2 45.87.77.15
2 45.138.55.61
2 156.106.214.52
1 91.177.126.188
1 87.65.25.98
1 45.138.55.62
1 185.51.192.63
1 185.51.192.61
1 185.111.204.220
1 162.159.200.1
Having more NTP servers serving the “be” zone would not lower the spikes, but it would make the spikes less frequent. I’d still aim for configuring everything so that even the spikes would not cause issues. At that stage it wouldn’t matter if you get those spikes once every hour or once every month.
As for your friend not getting the spikes, they seem to be getting around 950 queries per second on average, ie. they are already getting “your level” spikes all the time. They’re probably included in the be.pool.ntp.org responses much of the time due to their higher net speed setting. I don’t think there’s a mystery in here.
Sorry, you do not get it.
We lack 100 servers in Belgium.
As such lower served servers get spikes, often real bad.
You can not configure a router to get less spikes, as spikes are like a DDOS attack, they happen.
The router has to respond, ignore/drop/reject, whatever, it still gets the request.
In Belgium we need 100 servers extra to make the load distributed evenly. We only have a few.
And we get hit hard because of that.
Homeserver spikes and commercial router = overload / French server no spikes and ‘serves’ a lot more. BUT between () the number is much higher…meaning? I do not understand those numbers.
When discussing NTP request bursts, please remember the annoying source I’ve mentioned before.
Early this year I briefly ran an NTP server in the BE/Europe/Global zones, netspeed=500Mbps. The steady state request rate was around 500 per second, but there are bursts of 10x to 40x
This graph shows the total request rate.
I separated out the bursts that appear to come from misbehaving Fortigate clients and saw this.
This graph shows only the Fortigate clients (The Fortigate bursts have a characteristic pattern, see this note)
That is exactly what I mean. And when using an e.g. Fritzbox doing NTP-IPv4-behind-NAT, it will reboot at worst or stop serving DNS requests.
Just because of those spikes.
OpenWRT handles them just fine. But the problem is, as soon as you set the speed anything higher then 512Kbps, I presume you are getting more EU-traffic and the spikes start.
I wonder when zones change, I see them listed, how how are they affected?
Would be nice to have a way to prefer and dislike zones, not meaning disabling them, but get less attention of the pool-dns te be requested.
As Steve measured, they are a problem in the BE zone with 100 servers short compared to other zones.
