Beta - bad scores

Server operators
#1

Hi,
I just got email that my server is scheduled for deletion from the beta NTP pool system. On beta, I have about -50 score form both monitoring locations, but from real site, I have score >15. I am the only one with this issue?
Why the same monitoring location is able to connect to my server in real pool, but not in Beta?

https://www.ntppool.org/scores/92.240.244.202
https://web.beta.grundclock.com/scores/92.240.244.202

#2

The logs says you are rate limiting – the beta system is doing a few more queries (all should be 2 seconds apart) when testing.

https://web.beta.grundclock.com/scores/92.240.244.202/log?limit=200&monitor=*

#3

There may be a network problem near your server; I’ve asked for a packet capture.

One NTP request results in multiple responses. This occurs for both the IPv4 and IPv6 server addresses. I’ve seen this from multiple client sites, so the duplicates are not related to the NTP pool monitors.

Furthermore I believe that the long-observed network packet drops are occurring.

#4

@ask I’m not doing any special rate limiting, so it must be default ntpd. Should I somehow change the “discard” config option? I have updated debian on the server about 2 weeks ago, now ntpd is version 1:4.2.8p10+dfsg-3+deb9u2.

@stevesommars http://92.240.244.202/newark_capture.pcap

I’m not sure about multiple responses to one request. For example the last one from the dump, I see 2 requests from port 35085 and 2 answers from my server and the 2nd answer got port unreachable from the remote machine…

22:00:05.509595 IP 139.178.64.43.35085 > 92.240.244.202.123: NTPv4, Client, length 48
22:00:05.509674 IP 92.240.244.202.123 > 139.178.64.43.35085: NTPv4, Server, length 48
22:00:05.511739 IP 139.178.64.43.35085 > 92.240.244.202.123: NTPv4, Client, length 48
22:00:05.511797 IP 92.240.244.202.123 > 139.178.64.43.35085: NTPv4, Server, length 48
22:00:05.612015 IP 139.178.64.43 > 92.240.244.202: ICMP 139.178.64.43 udp port 35085 unreachable, length 84
#5

Here’s what happened, based on the the two tcpdumps. I’ll list the Newark side below, times are UTC

  1. NTP monitor sent one request
    2020-05-20 20:00:05.453937 139.178.64.43 -> 92.240.244.202 NTP 90 NTP Version 4, client

  2. That request was duplicated in the IP network between the monitor and your NTP server (more below).
    There are now two requests en route to your server. Both requests have the same NTP “Transmit time stamp.”

  3. First response arrives and NTP server responds:
    2020-05-20 20:00:05.555389 92.240.244.202 -> 139.178.64.43 NTP 90 NTP Version 4, server

  4. NTP pool monitor software closes UDP port 35085

  5. Second request arrives and NTP server responds
    2020-05-20 20:00:05.557512 92.240.244.202 -> 139.178.64.43 NTP 90 NTP Version 4, server

  6. Since the NTP Pool monitor has closed the port the incoming packet is rejected:
    2020-05-20 20:00:05.557535 139.178.64.43 -> 92.240.244.202 ICMP 118 Destination unreachable (Port unreachable)
    It might be easier to see this by comparing tcpdumps; that’s why I requested a sample pcap from your side.

I have some extra NTP probes running now, so this would be a great time to collect the tcpdump.

Tentatively I’ve identified the duplicates as originating from “Lightstorm”; I’m unfamiliar with this ISP.

Steve Sommars

#6

Ok, I have started another tcpdump now.
Lightstorm is the serverhousing ISP where this server is physically located. We have entire subnet 92.249.244.128/25 from them.

#7

Sorry, I have found old rule in iptables dropping NTP packets from IP if more than 4 arrive in 4 seconds. I have now changed it to 8/8. Let’s see if it will help.