Disable monitoring emails for a single server

Server operators
21

Have you checked if your firewall keeps track of the connections to/from the clients? There is typically a limit of how many such connections can be tracked. Optimally, you should not store connection state for NTP queries and their responses at all.

I’m not familiar with how OpenBSD does these things, but on Linux “conntrack -L” should show the current connection tracking table. On OpenBSD it might be “pfctl -s states”. “pfctl -s info” and its state-limit (“configured state limit was reached”) row might also be informative.

There should be at most a handful of connection tracking entries for UDP/123, possibly for your own NTP server’s queries to upstream NTP servers. You should not be seeing hundreds of connection tracking entries for regular NTP clients. Otherwise there’s a risk that the connection tracking limits get exceeded at some point.

You mentioned that you have set up similar pool servers elsewhere, but I think it is possible that those other servers have a lower query rate that does not exceed the connection tracking limits. There’s lots of NTP pool traffic in Singapore.

Edit: Overnight ping might not show anything useful. As can be seen from the linked graphs, there seems to be packet loss to your server between 02:00 and 15:00 UTC, probably correlating with the NTP traffic levels. During quieter times the ping will likely work fine.

22

Yeah, that was a good point. I keept it running a little bit longer.

--- sgh.miuku.net ping statistics ---
198045 packets transmitted, 198045 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.146/0.310/29.732/0.123 ms

So far, this looks good.

pfctl -s info   
Status: Enabled for 61 days 13:17:53             Debug: err

State Table                          Total             Rate
  current entries                    94387               
  half-open tcp                          0               
  searches                     35798503936         6731.2/s
  inserts                       5509307610         1035.9/s
  removals                      5509213223         1035.9/s
Counters
  match                         7245796538         1362.4/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                              13245            0.0/s
  normalize                            224            0.0/s
  memory                        1736048197          326.4/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                            266            0.0/s
  proto-cksum                          136            0.0/s
  state-mismatch                      7925            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  translate                              0            0.0/s
  no-route                               0            0.0/s

The limit is never reached. The firewall config has more than enought room for incoming requests. There is a Hetzner Firewall with a very small connection limit, but i have that disabled. Never used it on my servers, because the limit is something like 40000 a minute or so. Completly unusable for an NTP or DNS server for example.

I have an idea what could be an issue here, i will report back once i checked it out!

23

While the numbers above do not point to any obvious problems, my recommendation for not keeping track of UDP/123 traffic states still holds. 6731 state table searches per second averaged over 61 days is still a significant number. At peak times there will be 2-3x more lookups per second.

From my limited understanding of pf configuration, I believe the pf configuration would need a pass configuration with the no state option, both for inbound traffic to your server’s UDP/123 port and for outbound traffic from your server’s UDP/123 port. You would probably want to use the quick option as well to speed up firewall rule processing, and also place the rule near the top of your ruleset. I’d suggest trying this at some point if you want to reduce some overhead.

1 Like
24

Yeah, but even 4-5 times more is still nothing. But i will keep it in mind.

That is not really needed anymore, the internal optimizer does take care of that automatically. The good old days of ordering the pf.conf are thankfully over on OpenBSD. Nut sure about Linux hehe.

I found out what the issue is. I talked to a friend at Hetzner and after debuggin the logged traffic for days i found something. Someone is abusing that server by sending it manipulated UDP packets forcing the server to respond to addresses that never requested an NTP response. I got logs from Hetzner proving this, and somewhere upstream a Firewall just blocks the traffic after noticing this for a couple of minutes.

That explains most of the symptoms and i am working on debugging this (thats why i keep the state currently). However, the only issue that is still not explained is why your temp server has ping outages to my server (btw, i still cannot reproduce them from my end).

Just so that you are up to date on the current status.
Greeting :slight_smile:

25

On second thought, state searches themselves are easy. They can be done in parallel. The inserts and removals, however, may be trickier. I read in an ancient article that on OpenBSD the states are stored in an AVL tree, a balanced binary search tree. If the situation is still similar, maintaining the states may become a bottleneck. It may be possible to update or search different tree branches at the same time, but it will require some sort of costly locking and synchronization.

If there is too much congestion waiting for a lock to the state tree, the kernel may think “I don’t have time to sort out this packet, I’ll just drop it. The sender will retry if it was important.” This might show up in network stack statistics. Maybe see “netstat -s | grep -i drop” ?

As for the forged source IP address for NTP queries, well, it is possible but nowadays that scenario is less likely. I wonder what kind of “evidence” Hetzner provided. If it’s ICMP packets saying “udp port xxx at host yyy unreachable”, it’s possibly just broken NAT near the client. Not intentional abuse but bad configuration. I see this all the time on my servers serving the SG and CN zones.

I kept wondering about the 198045 echo requests you sent to my temporary server and the 198045 responses my server sent back, because at the same time there was certainly packet loss when pinging to the other direction. But then it hit me – it’s those connection states again.

You most probably pinged my temp server at the normal 1 second intervals, but my usual ping testing does the same at only 10 second intervals. With 1 second intervals the connection state does not expire, but it might with a 10 second interval. Meaning that every 10 seconds the state would need to be re-established and subsequently pruned.

To test this theory I set up additional testing using ping with a 1 second interval to your server. I’m pinging from two different servers at UpCloud’s Singapore DC:

10 second interval, 1 second interval

As should be clear from the graphs, pinging with 1 second intervals fares much better. If you want to reproduce the 10 second interval pings, this is the script that I use in my MRTG setup:

#!/bin/sh
PING="/bin/ping"
ADDR=$1

while true
do
	DATA=`$PING -n -c30 -i10 $ADDR -q | sed 's/+1 errors, //'`
	LOSS=`echo $DATA | awk '{print $18 * 10 }' | tr -d %`
	echo $LOSS
	if [ $LOSS = 1000 ]
	then
		echo 0
	else
		echo $DATA | awk -F/ '{print $5 * 10 }'
	fi
	sleep 10
done

You should be able to reproduce the results from pretty much anywhere with this script.

Maybe this is finally enough evidence for you to consider getting rid of tracking connection states for NTP requests :slight_smile:

26

I don’t think that’s correct. The memory counter shows that you ran out of memory 1.7E9 times.
The number of states is also close to the default of 1E5, so I’m pretty sure a lot of packets were dropped because there was not enough memory for creating new states.

27

How do I enable monitoring emails in first place? I have never received any so far.

1 Like
28

Good for you! The notification emails are sent automatically whenever the server’s score drops below some threshold, maybe -15 or so. If your server is working fine, you won’t get these nag emails. There is no setting to enable or disable these notifications.

2 Likes
29

Hi @leo, have you had an opportunity to consider going connection tracking free yet? Although I can’t guarantee that it would solve these problems once and for all, I have reason to believe that not tracking NTP traffic would help. The reason being these two ping graphs:

Ping with 10 second intervals:

ping10s
ping10s1010×484 28.7 KB

Ping with 1 second intervals:

ping1s
ping1s1010×490 27.7 KB

I think it is worth investigating why ping with 10 second intervals gives an average of 7.5% packet loss, while ping with 1 second interval gives only an average of 0.1% packet loss. Connection tracking might be one reason.

30

Do note that the traffic levels in Singapore are much higher than in many other zones. A server configuration that may have worked sufficiently well elsewhere (e.g. Germany, Finland) may run into problems in Singapore.

I believe OpenBSD’s pf keeps track of bidirectional UDP connections for 60 seconds by default. It can naturally be adjusted to some other limit as well.

Considering this, I captured 60 seconds of incoming IPv4 NTP traffic on my Singapore NTP server today at around 14:00 UTC. This pcap file was around 112 MB. My netspeed setting in the pool is 6Mbit/s, YMMV. Analyzing this pcap file I found:

1107517 requests in total
18459 requests per second (average)
498572 unique sourceip+sourceport pairs
367385 unique ip addresses

My stats say that a typical peak 5 min average for my server might be around 25k requests per second (or more), so it might not be entirely unreasonable to scale those numbers by a factor of 25000/18459 = 1.35.

In any case, if we take the original numbers and if I were to track the connections, I would need to be prepared to track ~500k connections. If some requests get dropped by rate limiting, those connections would not get the longer-lasting bidirectional status and those connections would get dropped from connection tracking earlier. However, I was interested in the upper bound of tracked connections, and in this case it was ~500k. Or maybe ~1M if we take the aforementioned scaling factor into account plus some headroom. Is this close to your “more than enough” limit?

31

Can you share this pcap with me?

I’m monitoring several NTP servers for anomalous clients. E.g., there are some noisy Fortigate(?) clients located in Lithuania.

Steve

32

Sure, feel free to grab it from here: https://miuku.net/tmp/1m_utc14.pcap.gz (30M).

33

Thanks. I don’t see any misbehaving Fortigate clients. There are several that send over 10 requests/second.

This is typical of other NTP servers I looked at.

Thanks for the data.

Steve

34

FWIW; I couldn’t resist and ran this pcap file through an old tool I once made.

Here’s the result:

plot
plot640×480 10.5 KB

It shows the distribution of the IP TTLs, which roughly reveals two things:

  1. Which OS-es are popular as clients?
    (default TTLs vary per OS, making them more or less identifiable - but it is no exact science)

and

  1. How far away (in terms of hop count) are your clients from the server?

Enjoy analyzing!

35

I agree with almost everything you are saying. I have keept it as it is for a reason, i changed the OpenBSD implementation of the lookup table to check if it could be speed up in the future. But right now, the new lookup algorythm does not improve the situation. Right now i am keeping an eye on it to investigate.

It’s very interresting to collect data in singapore, because its so different from Europe. Much more rogue clients and broken packages. Dealing with this is so much better than simulated packages in a dev environment.

The question is, should some rate limiting be applied? Or should all clients get a free pass to do what they want? Or is, for example a 500 request/second limit a good way of dealing with certain clients?

What are your thoughts on this?
Greetings
Leo

36

Me thinks it depends mostly on your situation if rate limiting is required.
An NTP server on a VM in a datacenter with unlimited traffic and sufficient bandwidth may require less ratelimiting than a small RPi with a consumer-grade router on an ADSL connection.
So really, it depends.

Personally i do have some ratelimiting for my NTP servers.

37

I see no valid reason for one client to send 500 NTP requests/second.
A rate limit is one approach.

There is another risk though. Assume that client C (a valid user) sends 1 request/minute to server S. Bad guy B sends a high request rate to server S using the forged IP address of client C. If server S rate limits the responses, then client C may never get a response. This is a type of denial of service attack.

38

To reduce the potential DoS effect of that, chronyd will randomly allow some packets through despite them exceeding a configured rate limit, with the rate of such a “leak” being configurable via the leak option to the ratelimit command.

2 Likes