Discussion of client traffic in BE (Belgium) zone

Server operators
1

Bas and I each have a client in the BE zone (actually BE+Europe+Global). Both provide good time accuracy & have scores of 20. Both have netspeed of 3Gbps. I’d like to verify that our clients get similar traffic. Bas suggested using this forum for the discussion, as others may be interested.

This discussion concerns the incoming traffic only. There are ongoing discussions concerning routers, IP tables, firewalls, etc. I’m reading those threads, but I have little to contribute.

I’ll start with my server:

image
image1173×446 8.51 KB

The average request rate is 1000-1500 /second, though there are significant outliers. Other averaging intervals (e.g., one second) look noisy. I expect to see a diurnal variation emerge as the tests continue.

I’d like to discuss abusive clients. One client, 194.78.232.126, sent 110+ requests/second for over a week. Some other clients send around 10 requests/second

Some people may remember my complaints about abusive Fortigate traffic This was due to an acknowledged bug. The vendor discussed the problem with me and the problem was fixed in an update. I started noticing similar NTP bursts coming from clients in Lithuania, some of which identify themselves as using Fortinet software. The vendors no longer answers my emails. How does this impact Belgium? One issue is that these other clients may burst simultaneously. E.g., here are 4 clients hitting my server at the same time.
2026-01-13 22:30:14 46.36.89.196 1145 FORT These clients are located in Lithuania
2026-01-13 22:30:14 46.36.90.231 1536 FORT
2026-01-13 22:30:14 46.36.90.211 1863 FORT
2026-01-13 22:30:14 46.36.90.216 3078 FORT

About 0.8% of my server’s traffic comes these Lithuanian bursters.

I’ve seen up to 15 abusive clients bursting simultaneously. I suspect other zones in Europe are seeing similar traffic.

2

I have also set 3Gbps…yet I have only 30Mbit upload.

See stats:

http://ntp1.heppen.be:90/

Funny is, I have at the moment 21K clients.

afbeelding
afbeelding1177×692 37 KB

But I do not have the speed to support this…I think.

My local network doesn’t suffer.

Abussive clients are not seen in this graph. However, I do believe many routers can’t handle xxxK IP’s and track them…their tables are just filled with rubbish.

I have been complaining for years and years…it’s the UDP-Timeout causing SHIT.

I run (almost) the same as Steven…but this is VDSL 100/30…should be maxed out. It’s not.

See:

afbeelding
afbeelding886×648 89.7 KB

192.168.1.50 is the NTP-server and NTP-monitor…see the real load, it’s nothing…sessions…yeah 33K!!

That is/was the problem… UDP-table-timeout…that kills your speed. Not traffic. BTW, the numbers you see are with 3Gbps netspeed set. :rofl:

3

For this client standard rate limiting should be enough, I guess.

46.36.90.231 and others nearby: I am seeing these clients, sometimes more than 1000 packets per second. It is totally blocked on my side, no reply packets sent.
I have 227 IP addresses in that situation, at this moment.
Out of that, 72 IPs from the 46.36.64.0/18 IP range, that is a dominant, big chunk.

5

I also see 46.36.90.231 in my BE server. In general these bursts from Fortigate(?) devices last for up to 10 seconds, sometimes with multiple clients bursting at the same time. There are quite a few routes, but the number of AS’s isn’t large. Here are some of them:

as13194
as15440
as202725
as21211
as24645
as33922
as43627
as49912
as8764

I have a lot of data on the Fortigate(?) clients from Lithuania. If there is interest, I’ll start a new thread on that.

As for 194.78.232.126 (continually sending 110+ requests/second), that is a different pattern. Do other people in the BE zone see persistent (e.g., lasting hours) abusive clients?

6

I had to set my homeserver back to 12mbit, 3Gbit was pushing it to the limits a little bit, as the page reported almost 1000packets/sec, but my modem reported 50Ksessions.

It did slowdown, but now I know a ‘normal’ setting won’t cause me problems anymore.

As for those bursts, yes I have seen them too, but comming from Belgian ISP’s as everytime I track them it’s Proximus/Telenet/Orange and they burst real big amounts at a time.
They are dropped by my ratelimit.

Beware Steve, your server is also Global set as far as I know, mine isn’t.

7

In addition to traffic rates & abusive clients, an NTP server & nearby equipment may be sensitive to the number of sessions. In some situations a session is necessary. A client utilizing a NAT address may need to have IP addresses and UDP ports mapped between inside and outside. A firewall may be used to restrict traffic. I’ll assume here that the device in question is an NTP server and that the sessions are present on the server and/or nearby firewall/router. Roughly how many sessions are involved?

For purposes of this discussion a session is defined by the client IP address and UDP port. [The NTP server address and UDP port are fixed]. Some device can display the number of sessions. They may even support configurable session timeouts. I took 24 hours of real NTP traffic from my 3Gbps BE NTP server and sent it through some simple session management scenarios. The scenarios are unlikely to match actual implementations, but the trends may be useful.

My BE server gets 1000-1500 requests/second. Some clients are well-behave, others are clearly abusive. When I run real traffic through the algorithm I see:
30-second session timeout: 20K-30K sessions
120-second session timeout: 100K - 150K sessions
600-second session timeout: 600K - 700K sessions

Also of interest is the session creation rate. [The session deletion rate will be similar.] This deserves a chart:

image
image1485×470 77.2 KB

Clients that poll every hour (or 15 minutes, etc.) are the main cause of the spikes. The above chart assumes a 600 second timeout. Shorter timeouts would result in higher new session rates.

You may be able to get information on the Client Configuration and Development from Chrony. E.g., see clientloglimit.

8

Well that is the problem, the number of IP’s that are left too long in the NAT-table.
I saw on my router at 3Gbps setting a clear >50K sessions, but the router can only handle 60K.

With limited IP’s this is never a problem, but with large number of IP’s the tables get exhausted and problems start in the local-network.
DrayTek confirmed this also.

But as I need Voip as well, I can’t reduce it under 30s.

Those abusive IP’s you see, I see them too and my ratelimit blocks them.

I’m very glad you confirm my previous problems! I think it should be told to home-users behind NAT that they have to reduce the NAT-IP-timeout.

Not many soho routers can handle this. On the MikroTik forum they also complained about this.

In my opinion routers should be aware that port 123 (NTP) should be able to set the limit to e.g. 5 seconds and drop it. Where it leaves it 30 sec (or more) for other services.

I tried to explain this to DrayTrek, they did not tell me to alter the firmware. Last year I told AVM Fritzboxes, they didn’t believe me and ignored my request.

How do we get modem-router firmware makers be aware of a very low timeout for NTP? As that solves all problems behind NAT.

Modem/Routers slowdown on heavy traffic
9

Assuming a software-based router, it takes CPU cycles to add/delete sessions. The lower the timeout, the more CPU cycles are spent in managing sessions.

I don’t know the relative CPU cost of adding/deleting a session versus the CPU cost of forwarding a couple of NTP packets.

From a functional NTP standpoint 10 seconds should be plenty of time.

10

I agree.

But the CPU isn’t the problem, it’s a Quad core very fast ARM CPU that can handle 950Mbps routing, way more then VDSL can handle as that’s max 300Mbit when bonded.
I only have 100/35Mbit, so CPU is doing nothing.
Memory isn’t an issue either, as it never tells me it’s out of it.

It’s just the NAT-state-table that runs out of space.

It’s my believe, DrayTek told me also, that it has to keep too much IP’s in a limited table.
The table is 60K NAT sessions, but another table is only 1024 entries.

The Active-NAT sessions table can only handle 1024 entries. I think it’s that table that can’t clear when the timeout is too high. Making it impossible to create new sessions, and thus also stopts resolving DNS.

Meaning it can, in my opinion, handle max 1024 IP’s for state at the same time. I believe this table is per/second. And the other 60K table is based on timeout or more ports?

Anyway, with timeout set at 180seconds I can only do a few sessions before troubles start, also on MikroTik and Fritzbox.
But when I reduce the timeout, I can do >50K sessions only to find some slowdowns, but it could be my ISP doesn’t like so much sessions.

My Chrony-server has no problem handling it all, nor does my local netwerk (1Gbit-cabled)…it’s the NAT tables. Not the modem-router CPU or memory.

This is my take on what is going on, as I have no problems and it’s set 6mbit Pool-speed, before 512K and troubles. Now, none. Only by changing the NAT-timeout to 30 seconds.

PS. the router has hardware-acceleration and it’s turned on. I can see in the router what sessions are accelerated and what not. About 50% are.

11

I don’t think that it will be possible to convince the vendors of bargain basement CPE that there are many customers who need to be hosting demanding UDP-based services. While what you describe is trivial to configure in the Linux kernel that most of them probably use, it costs them money to [pay developers to] expose those settings and document them, on an ongoing basis. Even putting you in conversation with an employee who understands what this is about is likely not going to make them any money.

I think it is likely going to fall to us to explain that any service with a high number of flows is going to be problematic without either avoiding NAT or tuning it.

Depending on how the connection is presented it is very possible that the CPE can be a customer-provided generic off the shelf device, for example if it is PPP-over-Ethernet all that you need is something with more than one Ethernet port. If not that then something that can be put in “bridge mode”. Anything to get the NAT off the cheap equipment. Then it is easy to find tuning guides for the firewall software used.

There’s also no NAT involved in IPv6 (unless one tries really hard to deploy such an abomination). :grinning_face:

12

Even if hardware acceleration is used in the user-plane (forwarding IP packets, doing NAT, etc.),
the acceleration configuration is likely to be done by the CPU. Would high reconfiguration
rates (e.g., from low timeouts) consume excessive CPU? We don’t have enough information to make an educated guess.

Looking at the distribution of intervals between NTP requests from a single IP, I see spikes around 64 and 1024 seconds.
There are many clients with shorter intervals between requests, though.

13

The CPU is doing nothing. That is the thing, it’s loaded to max 10% of capacity when 3Gbit poolspeed is set.
And when I tried a few hours yesterday with UDP timeout-10sec, it never caused me problems.
Not a single moment.

I would say, if you have a home-internet connection, run an NTP-server, put it in the Belgian pool and try to run 3Gbit, but 512K is enough to make it run into troubles after a few hours.

You have to try at home with NAT, then you will see what happens. It’s the NAT-tables that crap out, nothing else. Not memory, not the CPU, not the NIC, not Chrony, not my Network etc.

Unless you try your own modem-router with NAT to do it, you will never see it. Rack-machines are not natted, they do not experiance it.

Trust me, I have tried anything, until I found my DNS-stopped resolving, then I checked UPD issues with NAT…

MikroTik - RB5009 tested and failed.
Fritzbox 7590 - tested and failed, has even been back to AVM to get replaced, replacement failed too.
Fritzbox 7530 - tested and failed.
Got a FritzBox 5690Pro…top of the line…guess what…it failed.
Draytek 2865LTE-4G tested and failed, 550 euro modem-router!
Draytek 2865AX newer and faster model, tested and failed.

Replaced, modem-part with all by Zyxel 400 DSL modem, modem worked, all routing failed.

Tried PfSense…failed. Tried OPNsense…failed, tried OpenWRT couldn’t get it to work, tried RouterOS X86 never worked.

Tried everything. UNTIL I lowered the UDP timeout-session-table setting, 30 secs works well but 10sec is even better, it never fills to the max.

As soon as I change the UDP-timeout it works well. My friend in Leuven also change that setting before and has no problems with the MikroTik ( @microchip8 ) and he has a 500Mbit fiber connection. Maybe he can set it to 360 sec’s for testing for you and see it fail, as it will (99.99% sure).

14

What did the pfsense failure look like?

I started using pfsense almost 20 years ago and haven’t found anything I like better

15

The problem is mostly that I do not know bsd. And I like to work Cli with midnight command to do my stuff.
It couldn’t find it in the package list.

But not to worry, the DrayTek works fine now.

16

Yeah, i concur.

Happy OPNsense user here. Box handles forwarding traffic for 2 servers and hosts chrony server for 4 others. 6x NL zone and 1x BE zone. Firewall states hover usually around 10,000. Firewall setting is”agressive”.

Breaks barely a sweat.