Using an NTP-server behind NAT, it works! After testing a lot. No not many routers/OS can handle it. Really not many

Server operators
1

Hi all,

You know me, have been fighting routers for a long time.
Finally cracked it. There is so much crap around, from opensource to expensive.
They all fail, and it’s so stupid they fail! It makes no sense.

BTW, I’m using OpenWRT now, it fails too by default!!! Yes it does. But here is what I changed:

I started to change the flood parameters, as they are bad:

/etc/sysctl.conf

net.netfilter.nf_conntrack_acct=1
net.netfilter.nf_conntrack_checksum=0
net.netfilter.nf_conntrack_tcp_timeout_established=7440
net.netfilter.nf_conntrack_udp_timeout=15
net.netfilter.nf_conntrack_udp_timeout_stream=180
net.netfilter.nf_conntrack_max=524288
net.nf_conntrack_max = 524288

This makes the tracking a lot bigger then the default. I never seen more then 60K.

Then the Firewall /etc/config/firewall edit it:

config defaults
         option input 'REJECT'
         option output 'ACCEPT'
         option forward 'REJECT'
         option synflood_rate '2500/s'
         option synflood_burst '5000'
         option flow_offloading '1'

the default is 25/s, not going to work!

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'NTP'
        list proto 'udp'
        option src 'wan'
        option src_dport '123'
        option dest_ip '192.168.1.50'
        option dest_port '123'

Last part for the firewall, make a rule:

config rule
       option src 'wan'
       option dest 'lan'
       option name 'NTP-no-tacking'
        list proto 'udp'
        option src_port '123'
        list dest_ip '192.168.1.50'
        option dest_port '123'
        option target 'NOTRACK'
        option enabled '1'

That makes tracking impossible. So it doesn’t take much table space.

Not much more you need to do.

At the moment it’s kicking a whopping 40K connections, no sweat at all.

Any NUC with 2GB ram will do.

afbeelding
afbeelding1684×2190 276 KB

Also important!

afbeelding
afbeelding1684×1204 118 KB

Enjoy. And I hope I didn’t forget anything. It works for me, and it can NAT a lot!

Most routers fail, no joke, a lot fail.

I’m testing…but it seems to handle just fine.

2

No issues so far.

afbeelding
afbeelding1177×692 42.5 KB

Connections…

afbeelding
afbeelding1537×1277 98.1 KB

I notice nothing on my local network. :+1:

96K connections :rofl:

3

I don’t recall having to do anything performance-wise for port forwarding on my opnsense router. Intel N150 I think, 250 requests per second.

4

That suggests you still have some connection tracking for NTP, possibly created by the server responses. The server traffic should not increase the number of connections at all. With nftables it needs two notrack rules, one for incoming direction, one for the outgoing direction, see the example in this post: Joining the pool kills my Internet - #21 by mlichvar

Use the nft list ruleset command to verify the rules generated from the OpenWrt firewall config.

5

Hi Mlichvar,

I have tried to implement your rules, but I’m probably to dumb for it, so complicated.
There must be an easy way to do it.

But it doesn’t really matter as my setup can handle the 3Gbit speedsetting, the router doesn’t run into troubles, just slows down a bit.
As I’m not using this setting for a day-to-day setting, I’m perfectly happy how it runs at 12mbit setting.

My head spins over all things I have tested and tried the last few weeks.
I leave it the way it is for now.

I will look into it someday…then I come back to that part.

6

I’m not testing with 250 but with spikes of >1500 a second.

All my problems started at less then 100 on commercial routers.

But now I’m pushing way beyond the limits I need.:grin:

7

This is good work. It would be nice to quantify the NTP request load. I recently added a server in the BE zone and monitor the traffic closely.

If the firewall or NTP server is stateful, the connection rate may be significant. A crude look at connection rate uses the number of unique clients seen each second. Unsurprisingly there are spikes on the hour, on the half hour. I see similar patterns in other regions.

image
image1166×441 5.07 KB

We’d like to know both the number of connections and the connection rate. That would require assumptions on the connection management strategy.

There are unusual traffic bursts in Europe. Here is one example from the same server (note the y-axis log scale):

image
image1170×440 4.83 KB

Usually the request rate is under 100/sec. For a ten-second interval it grew to 14,000/sec. This spike was due to one client, which I believe involves Fortigate software.

If Bas does additional testing more information on the offered load would be appreciated.

8

The OpenWrt firewall config (fw4) doesn’t support stateless NAT, at least I don’t see how it could be specified. So you need to load it in the nftables syntax. An easy way to do that is to create an *.nft file in /etc/nftables.d with new chains (named differently than the chains used by fw4).

For example:

# /etc/nftables.d/90-stateless-ntp.nft

chain ntp_prerouting {
	type filter hook prerouting priority raw;
	iif eth0 udp dport { 123 } ip daddr 100.100.100.100 ip daddr set 192.168.123.99 notrack
	iif br-lan udp sport { 123 } ip saddr 192.168.123.99 notrack
}

chain ntp_postrouting {
	type filter hook postrouting priority raw;
	oif eth0 udp sport { 123 } ip saddr 192.168.123.99 ip saddr set 100.100.100.100
}

Where eth0 is the WAN interface of the router with IP address 100.100.100.100 and br-lan is the LAN interface where is the NTP server with IP address 192.168.123.99.

In the firewall config there needs to be a rule allowing the incoming packet with the modified destination address to get through to the server, e.g.

config rule
        option src 'wan'
        option dest 'lan'
        option dest_ip '192.168.123.99'
        option dest_port '123'
        option proto 'udp'
        option target 'ACCEPT'

Remove the redirects and rules you have added before, restart the firewall or router, and you should see the number of connections drop to near zero.

9

Hi Mlichvar,

Well I set a traffic-rule that seems to work, see Status->Firewall it is catched 8.2MB:

afbeelding
afbeelding1179×138 5.69 KB

The rule itself is set under Network->Firewall->Traffic-rules->add:

afbeelding
afbeelding1264×234 22.5 KB

It seems to work, but does it do anything? Have to test as my active-connections dropped with >10000 to less then <3000

Update: I added another the other way around…

10

Oh yes they are working…WAN:

afbeelding
afbeelding900×576 30.3 KB

afbeelding
afbeelding1334×128 5.01 KB

LAN-No-track just makes the server stop responding.

11

NTP uses port 123 for everything.

I think you’d need to change your No-track-NTP-WAN rule. Now it’s apparently:

From wan, port 123
To lan

when it should really be

From wan
To lan, port 123

The source port (NTP clients) can be either 123 or something else entirely, but your NTP server’s port is always 123. The other direction (No-track-NTP-LAN) seems to be correct.

12

Changed it, seems to do a bit more. Thanks.

13

It does matter if you want to achieve “no tracking” for all NTP traffic (which you do).

YMMV, but on one of my servers the source port is 123 only in 11% of requests. As I understand it, your current No-track-NTP-WAN rule will apply the “notrack” rule only for those requests where the source port is 123. If you used “any” as the source port and 123 as the destination port (to limit the notracking to NTP traffic only), you’d be “notracking” all your NTP traffic.

Yes, if your NTP server listens on your LAN on some other port than 123 then the rule would need to be changed accordingly, but based on your other config snippets above you’re not doing that (ie. your NTP server on LAN uses the standard 123 port), so the point is somewhat moot.

14

Got it mate.

Very confusing this stuff.

But we will get there, one way or the other. :+1:

Did that and the server won’t respond anymore.

This works:

afbeelding
afbeelding900×576 29.7 KB

15

NTP request packets (those coming to your NTP server) will have the destination port as 123 and the source port can be anything. The response packet will swap it around, so the source will be 123 and the destination whatever was in the request source port.

So the rule for incoming packets (requests) should have 123 in the destination and the outgoing rule should have 123 in the source port.

It looks like that is what you have now.

16

It is possible that you have some existing rule that allows bidirectional traffic for established flows (ie. tracked connections). This is normal. This lets you e.g. browse the web from the other computers on your LAN.

Removing the tracking for NTP packets made that “allow established connections” rule not care about NTP traffic. That might actually be good progress.

If you no longer have tracked connections, you may need to add explicit rules for allowing traffic from WAN to be forwarded to your LAN NTP server port 123, and for allowing traffic from your LAN NTP server port 123 to be forwarded to WAN.

17

When packets are marked not to be tracked, stateful DNAT, the only DNAT that seems available via the GUI, will not work on them. You’ll have a better chance following @mlichvar’s suggestion, using stateless DNAT.

18

As I stated before, I have no clue what he’s doing, as his ports are far different then mine.

root@OpenWrt:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP qlen 1000
    link/ether 00:e2:69:9e:33:5e brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:e2:69:9e:33:5f brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:e2:69:9e:33:60 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:e2:69:9e:33:61 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:e2:69:9e:33:62 brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:e2:69:9e:33:63 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2e2:69ff:fe9e:3363/64 scope link 
       valid_lft forever preferred_lft forever
8: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:e2:69:9e:33:5e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fda9:9e:825d::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::2e2:69ff:fe9e:335e/64 scope link 
       valid_lft forever preferred_lft forever
9: eth5.10@eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:e2:69:9e:33:63 brd ff:ff:ff:ff:ff:ff
10: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc noqueue state UNKNOWN qlen 1024
    link/ppp 
    inet 185.142.225.68 peer 109.233.104.27/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever
    inet6 fe80::e5e2:bfd2:b3ea:c6dc/128 scope link flags 02 
       valid_lft forever preferred_lft forever

WAN IP = 185.142.225.68
LAN Net = 192.168.1.0/24
NTP IP = 192.168.1.50

It’s way off his explanation. Please help. I do want to fix this once and for all.

19

Under the snippet, he described the adaptations to be made:

100.100.100.100 → your external IP address, 185.142.225.68
192.168.123.99 → the internal IP address of your server, 192.168.1.50
eth0 → the name of the WAN interface that has the external IP address, i.e., pppoe-wan

20

Won’t work.

As the WAN is routed via Vlan10…whatever port.

It used to be so simple, today it drives me nuts.

BTW, the port for WAN cable connection is eth5, but then directed to vlan10

eth0 is the local lan, I think. But then bridged?

Sorry but I do not understand networking anymore, people who invented this must have been on serious LSD or worse.