Joining the pool kills my Internet

Pool Development
1

Ok so I tried every possible way to make my server pool member, but every time I join the server my whole internet crashes !

Then, I must scheduled my server for deletion in order the system to remove my server.

I am using Linksys E4200 with DDWRT(I tried with different router it is the same).

From what I can see is there is too much connection and the router’s connection table is getting full and then simple begin to drop packets.

I even tried with different speed on pool server management but it is still the same, even with the lowest one.

Its like every opened UDP connection reaming open, I have tried every possible way to solve this, but without luck, and I am out of ideas.

Have someone dealt with this or similar issue ?

2

EDIT

I did not know how I missed this obvious settings. I am getting sleepy I gues :slight_smile:

image
image.png722×179 4.65 KB

I think with this I will solve the problem, but tommorow, now I am to tired to think.

I will report the results

EDIT:

Unfortunately timeout settings do not help. I am out of options.

3

You are using connection tracking. Turn it off for UDP/123 traffic in both directions (just doing that in one direction isn’t enough) on any stateful device in your traffic path - this includes your system firewall if you have one, and almost certainly your router. If you are using NAT (this includes port mapping), don’t - most forms of NAT imply connection tracking on the device doing that translation. If your router isn’t capable of excluding NTP traffic from connection tracking (most typical home routers can’t easily do this), then you will need a new router.

The reason that this is a problem is that the NTP pool is extremely diverse, and you’ll see a very large number of client IPs in a short period of time. NTP is stateless (one request packet, one reply packet, that’s it), but to most forms of connection tracking, it looks like a newly created connection flow that ought to be tracked. This means CPU usage (watching for reply packets), RAM usage (storing the state of the ‘connection’ for each client IP), and more CPU usage (timing out connection state and removing it from memory). While that’s acceptable for a small number of clients, a pool NTP server can easily see requests from hundreds of thousands of unique client IPs within a short timespan, which will cripple many connection-tracking devices due to insufficient resources to handle the load (and sometimes poorly optimised software as well).

4

Yes I realized that, but unfortunately UDP timeout it did not help.

It was suggested by user Miroslav Licvar on group com.protocol.time that I try followoing rules for UDP tracking:

iptables -t raw -I PREROUTING -p udp -m udp --dport 123 -j CT --notrack
iptables -t raw -I OUTPUT -p udp -m udp --sport 123 -j CT --notrack

But it did not work also. Maybe these iptables command do not work on DDWRT.

I am out of option, for now I cannot join the pool.

5

Yes I realized that, but unfortunately UDP timeout it did not help.

That’s because UDP timeout is intended to adjust the timeout for removing tracked flows. It’s not the right setting to adjust for solving your problem.

It was suggested by user Miroslav Licvar on group com.protocol.time that I try following rules for UDP tracking…

Those rules are intended for use on your NTP server, not on your router. To properly exclude your NTP traffic from connection tracking, you will also need an additional rule in the PREROUTING table with --sport 123 in order to catch the reply packets from your server.

I notice that those rules do not contain your server’s IP address at all - that’s OK, but do bear in mind that omitting the IP means you are turning off tracking for all NTP packets, rather than just NTP traffic relating to your server.

…But it did not work also.

Can you clarify what you mean by that? What, specifically, happened (or didn’t happen) to indicate that there was an issue with those commands?

6

When I commit iptables rules in DDWRT router I get this:

image
image.png941×448 8.56 KB

Commands are not showing any errors but they are not in iptables list. Maybe they cannot be issued in DDWRT version of the router.

7

What does iptables -t raw -L show?

If you run echo $? immediately after each iptables command, what is the output?

8

root@life-router:~# iptables -t raw -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
CT udp – anywhere anywhere udp dpt:ntp CT notrack

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
CT udp – anywhere anywhere udp spt:ntp CT notrack

9

That output means that those iptables rules were applied successfully. If you add an additional rule to the PREROUTING chain to disable tracking for the reply packets (and assuming you aren’t attempting to use NAT for the NTP traffic), that should be sufficient to resolve the issue with your router.

10

@erayd , I’m attempting something similar with DD-WRT.

The udp port 123 connection tracking is slowing the router and filling the connection table to the point where DNS is not working or delayed for several seconds attempting to get udp out and back in.

I’ve added this iptables line manually as root at the dd-wrt bash prompt:

iptables -t raw -I PREROUTING -p udp --dport 123 -j CT --notrack

But the results are a mix of success and unintended consequences. Afterward:

  1. udp connection tracking stops on the router (this is good)
  2. ntp traffic stops arriving at the linux NTP server within the router’s protected subnet (this is not good)

I’m not using NAT, but have about 5 port range forwarding rules in the DD-WRT web interface. It seems the rule to stop udp connection tracking also stops port forwarding.

Is it possible to stop tracking and still enable port forwarding?

I don’t really want to manage iptables rules manually. The web interface is so much easier.

11

I can’t speak to DD-WRT specifically, as I don’t use it. However, more generally, in most situations you do need connection tracking enabled in order for NAT to work.

For clarity, as you seem to be saying you use port forwarding but not NAT - port forwarding is a form of NAT.

12

Yes, perhaps. The implied Network Address Translation (NAT) is from the router’s external, WAN-side IP to an internal LAN-side IP. I’m not modifying that from default.

What I am doing is specifying some ports to be forwarded and all the others blocked. That dialogue box is certainly with in the NAT/QOS tab in DD-WRT’s web interface.

It sounds like there’s not a clear solution here.

The connection tracking seems to be an integral part of NAT in it’s various subtleties, at least within DD-WRT. Connection tracking is also limiting max NTP requests per second.

13

The implied Network Address Translation (NAT) is from the router’s external, WAN-side IP to an internal LAN-side IP.

That’s not just “implied” - that is what is commonly known as “destination NAT” (often abbreviated as DNAT). It rewrites (translates) the destination IP address of the inbound packet, and possibly also the port.

I’m not modifying that from default.

On most home routers, the default is to track everything, because that’s what most people want. DD-WRT is no different in that respect.

The connection tracking seems to be an integral part of NAT in it’s various subtleties, at least within DD-WRT.

Connection tracking is used by NAT in order to figure out what the source IP / port of the reply packets needs to be rewritten back to prior to forwarding them on. This is why, in most cases, disabling connection tracking will break NAT. Often you’ll see that the replies still go out, but with the wrong source - the router doesn’t know it’s supposed to be rewriting the source address for them, and they then get dropped (either by your own router or somewhere upstream) because they have a source address in a private address space, which should never appear on the public internet.

ntp traffic stops arriving at the linux NTP server within the router’s protected subnet (this is not good)

If packets aren’t arriving at the server, the router may be trying to optimise untracked packets and using a cut-through routing path which excludes NAT entirely.

It sounds like there’s not a clear solution here.

The clear solution is to serve NTP from something that has a public IP assigned to it directly, and does not require NAT.

If that isn’t an option, a partial workaround is to drop the packet rate to something much lower, to limit the number of unique clients your router is likely to see in any given time period. You may also wish to look into the tracking timeout for UDP flows. Given that NTP uses only a single packet request and reply, there is absolutely no value gained from having that traffic clutter up the tracking tables once the reply is sent.

14

Connection tracking is not necessary to redirect a port. You just need to change the address in both directions, i.e. one -j DNAT rule and another -j SNAT rule. You should also specify the input and output interface with the -i and -o options as appropriate. I’m sure there are plenty examples out there.

EDIT: Hm, per this post it is really not supposed to work. I probably confused it with something else.

15

@mlichvar , thanks for the idea. That post indicates it should not work, just as you said, but I’m surprised at this lack of configurability. It’s a bizarre limitation, especially with UDP. I’m still considering options. It’s not critical this problem be solved but it’s intriguing enough to keep investigating.

16

@erayd ,

I’ve already dropped the connection time on UDP packets to 30 seconds and increased the max number of tracked connections. NTP is working well with about 80 to 200 packets per second but I was hoping to dial up the Pool Speed Setting. It’s an interest but not an important need. It’s just not possible without substantial change, like getting another IP or changing routers or changing router firmware.

Serving NTP from a public IP address directly is plausible, but still needs to provide internet to the LAN side. These are some options:

  1. Run Chrony on the router, from DD-WRT (don’t think this is possible)
  2. Replace DD-WRT with OpenWRT on WRT-1900AC hardware and run Chrony
  3. Put OpenWRT on a raspberry pi hardware
  4. Run pihole and add Chrony to the pi

Option 2 might make sense as long as the GPS with PPS can be detected properly. It’s a little involved tho. It’s working just fine right now, at low’ish packets per second.

17

You could also consider switching the router to nftables, which might be able to perform a stateless NAT specifically for NTP packets.
https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT)#Stateless_NAT

I have not tried it.

OpenWrt and DD-WRT don’t seem to support nftables in their configuration tools, so you would need to disable the iptables support, install the nftables kernel modules, and configure the firewall and NAT manually.

Running the NTP server on the router or the Pi (assuming you turn it into a router with a USB ethernet?) might be easier. I think they should be able to handle about 20kpps.

1 Like
18

@erayd ,

If I get Chrony working on an RPi4 operating as a wired router, it’s still Linux and still, presumably will have iptables functioning.

Is the same conntrak connection tracking problem going to remain?

How do commercial grade routers handle so many more simultaneous connections? What’s the primary difference that allows improved performance by commercial over consumer grade devices?

19

@mlichvar , this sounds involved, but stateless NAT seems to be what’s needed. It’s a good insight and possible solution.

1 Like
20

Yup. Ultimately, unless you can avoid tracking state for those flows, you’ll run into the same limitation. The only difference will be how much it can handle before it hits the limits of what its CPU and memory can cope with.

Can you be a bit more specific with your question?

As a general rule, transit routers that handle a large number of flows don’t have any state tracking at all enabled for forwarded traffic. They just shunt individual packets around between interfaces, which doesn’t require much at all in the way of resources, and thus scales up very well indeed. Occasionally you do run across a provider which accidentally provisions a router with connection tracking enabled, and you can severely impact performance on that router by running a pool server on your connection.

Things like CGNAT gateways do normally track state, but will be a fair bit beefier in terms of resources than your typical residential router. They still have the same limitations, but a lot more resource headroom to work with. Plus, customers generally don’t try to run things like NTP pool servers behind CGNAT, so that helps too :slightly_smiling_face:.