Monitoring station unable to query NTP servers

For years now I’ve had a problem that for some reason my NTP servers work fine via one Internet provider but not the other one. So my guess is that somewhere between the monitoring station and my servers port 123 is being blocked. Right now I set up 2 servers for testing, one works the other does not, both can be reached by the monitoring station just fine:

provider 1 works:

Traceroute to 200.59.21.5
 1  *  *
 2 (169.254.74.0)  0.132  0.159
 3 (10.253.4.207)  0.121
 3 (10.253.4.209)  0.212
 4 (10.253.4.147)  9.348
 4 (10.253.4.149)  1.052
 5 (62.115.179.210) AS1299  0.439
 5 nyk-b4-link.ip.twelve99.net (62.115.180.4) AS1299  0.441
 6 nyk-bb1-link.ip.twelve99.net (62.115.114.178) AS1299  1.253  1.133
 7 (62.115.136.201) AS1299  7.036
 7 rest-bb1-link.ip.twelve99.net (62.115.141.244) AS1299  7.790
 8 mai-b2-link.ip.twelve99.net (62.115.120.177) AS1299  31.769  31.737
 9 navega-svc070684-ic356808.ip.twelve99-cust.net (62.115.56.165) AS1299  32.312  31.536
10  *  *
11  *  *
12 (186.176.7.74) AS262197  81.198  81.331
13 www.fratec.net (200.59.21.5) AS262149  81.141  71.528

provider 2 does not work:

Traceroute to 200.59.20.230
 1  *  *
 2 (169.254.74.0)  0.171
 2 (169.254.74.1)  0.125
 3 (10.253.4.205)  0.107
 3 (10.253.4.207)  0.084
 4 (10.253.4.149)  1.222
 4 (10.253.4.147)  0.669
 5 (62.115.179.210) AS1299  0.344
 5 nyk-b4-link.ip.twelve99.net (62.115.180.4) AS1299  0.349
 6 nyk-bb1-link.ip.twelve99.net (62.115.114.178) AS1299  1.317  1.334
 7 rest-bb1-link.ip.twelve99.net (62.115.141.244) AS1299  8.742  7.133
 8 boca-b2-link.ip.twelve99.net (62.115.123.29) AS1299  32.434  32.461
 9 (62.115.11.225) AS1299  32.261
 9 asurnet-svc080563-ic370378.ip.twelve99-cust.net (62.115.149.31) AS1299  32.474
10 (69.79.102.1) AS23520  32.700
10 ae2.brx-mx2020-2.boca-raton.fl.usa.cwc.com (69.79.100.5) AS23520  32.448
11 (69.79.102.1) AS23520  32.467
11 (69.79.106.49) AS23520  52.216
12 (69.79.106.49) AS23520  52.502  52.001
13  *  *
14 111-72-179-186.ufinet.co.cr (186.179.72.111) AS52468  69.117  68.831
15 (200.59.20.230) AS262149  69.357

Now using wetwiz.net both look ok:

NTP Server Test for 200.59.21.5

NTP Server Test for 200.59.20.230

But if I try to add the second one to the pool I get

200.59.20.230

Could not check NTP status

So, I need help sorting this out, I need to pinpoint the problem so whoever is responsible can fix it. If anyone has an idea how to find the problem, please let me now. Thanks

I checked NTP reachability from a couple of my clients. NTP worked for the client hosted on Comcast, but did not for the client hosted on AT&T. I checked traceroute (mtr).

It is important to probe using multiple UDP destination ports, in particular NTP (UDP port 123) and some other UDP port.

NOTE: mtr and traceroute options may vary.
I used:
mtr -n --udp -P 123 200.59.20.230
and
mtr -n --udp -P 124 200.59.20.230
The results showed blockage, but some nodes did not respond to traceroute. (I sent the details to Profile - jfrater - NTP Pool Project).

Next I logged onto monsjc2, the NTP Pool monitor and ran the same commands and also emailed the full traceroute to jfrater . To highlight the difference.

Port 124
Loss% Snt Last Avg Best Wrst StDev
7. 64.86.160.12 61.1% 18 1.3 1.6 1.2 2.2 0.4
8. 193.251.143.55 50.0% 18 1.6 1.7 1.3 2.2 0.3
9. 193.251.242.2 0.0% 18 74.6 42.3 1.4 75.0 37.2
10. 193.251.242.2 0.0% 18 74.6 74.8 74.4 75.5 0.2
11. 193.251.254.124 0.0% 18 74.9 101.6 74.9 123.6 24.2
12. 200.59.20.230 0.0% 18 122.3 122.5 121.7 124.1 0.7 <<< NTP server

Port 123

  1. 64.86.160.3 30.0% 10 1.7 1.7 1.2 2.3 0.4
  2. 64.86.160.3 10.0% 10 1.8 3.3 1.2 15.3 4.5
  3. 193.251.242.2 0.0% 10 74.5 45.7 1.4 75.2 37.7
  4. 193.251.242.2 66.7% 10 75.1 74.7 74.5 75.1 0.3
    11 no response

This appears to be an NTP specific block close to 193.251.254.124 (Orange - OINIS)

Occasionally the Port 123 trace shows
1220 2023-02-26 01:42:12.761014 193.251.254.124 → 147.75.202.162 ICMP 110 Destination unreachable (Communication administratively filtered)

This suggests the blockage is intentional.

TLDR: AT&T blocks packets coming from its network whose source port is 123 on IPv4.

2 Likes

AT&T doesn’t always block 123 it seems to happen when “congestion” rules are put into place. Overnight I can have a perfect 20 then at 5:50 am the monitor can’t reach me. It varies over the day and also varies over the weekend. I also see the congestion rules come into play where they have blocked 5xxx port ranges and I cannot access my NAS services from the public internet.

1 Like

I’m seeing the same issue it appears on my NTP server. It started about 24 hours ago. The monitoring stations fail sometimes and I have been removed from the pool. I called my ISP and they don’t block or use any congestion rules. I have checked from some external locations and everything looks good. I have checked everything on my end… router, switches, tried different NTP servers on my LAN etc… ntpsec or ntp, it doesn’t matter. My IP is: 68.69.165.2 (time.nc7j.com). If someone would like to take a peek on your end. Below is a image showing the start time of the problem yesterday. mg

image

Seems like you are not reachable for the Pool.

https://trace.ntppool.org/traceroute/68.69.165.2

Traceroute to 68.69.165.2
 1  *  *
 2 (169.254.74.1)  0.114  0.109
 3 (10.253.4.211)  0.186
 3 (10.253.4.205)  0.140
 4 (10.253.4.147)  0.245  0.256
 5 nyk-b4-link.ip.twelve99.net (62.115.179.210) AS1299  *  0.623
 6 nyk-bb1-link.ip.twelve99.net (62.115.114.178) AS1299  1.820  1.913
 7 chi-bb1-link.ip.twelve99.net (62.115.118.149) AS1299  18.066  18.006
 8 den-bb1-link.ip.twelve99.net (62.115.115.76) AS1299  40.796  40.715
 9 den-b2-link.ip.twelve99.net (62.115.127.67) AS1299  40.927  40.874
10 den-b3-link.ip.twelve99.net (62.115.139.206) AS1299  41.930  41.658
11 salt-b2-link.ip.twelve99.net (62.115.132.207) AS1299  50.931  50.952
12  *  *
13 xmission-lc.10gigabitethernet6-14.core1.slc1.he.net (65.19.138.174) AS6939  70.035  70.262
14 mar2.core.xmission.net (166.70.1.51) AS6315  73.073  71.993
15  *  *
16  *  *
17  *  *
18  *  *
19  *  *
20  *  *
21  *  *
22  *  *
23  *  *
24  *  *
25  *  *
26  *  *
27  *  *
28  *  *
29  *  *
30  *  *

Maybe your ISP is blocking the NTP Pool Monitoring IP’s because they ratelimited it in a way ?
Is that a dedicated server or which device is that?
Home Hosted?

I ran a quick test and believe that Telia (probably) is rate limiting port 123.
This is fairly common.

Hi, I block ICMP packets at the router. The isp doesn’t block any traffic or have any traffic / load rules in place… called and checked.

This is a static ip with a valid reverse dns host name I own. Nc7j.com

I host some simple websites, I.e www.syracuseutweather.com you should be able to access that site. I also host some telnet related hobby endpoints and have dozens of i users connected 24x7. No issues with those users.

I even disabled my firehouse IPsec bad actor IPsec list blocking. No difference.

Still scratching my head. A lot of the request coming in on port 123 work fine. Some ntp pool monitors are succeeding. Try the ntp server test. Sometimes I get 3 successful tests and sometimes a few succeed and others fail.

Thanks for the help. Mg

Steve, that makes the most sense… somewhere in the middle from the monitor to my isp. Is there any recourse with them that you know? Mg

I have tried for years to get Zayo & Telia to even acknowledge that they rate limit NTP.
No response from either.

Makes no sense in my opinion that telia is rate limiting port 123.
first time i ever hear that.

Please see my article

NTP blockage was a major factor in the NTP monitoring system redesign (multiple distributed monitors).

Well, I’m back in the pool but I have never seen a 20 score on a server with a monitoring graph that looks like this. :sunglasses: Clearly, IP routes from some of the monitors are not making to me and everything is fine on my side. FWIW, here are the ntpviz reports for the last 24 hours on the ntp server I’m using (one of 5 :slight_smile: ) http://time.nc7j.com/ mg

image

1 Like

It’s pretty curious that it completely cleared up in the last 3 days!

https://www.ntppool.org/scores/68.69.165.2

Hi Ask, yes… I have no idea why. I did drop the bandwidth use down. I have been out of town off the grid for 5 days now. I’ll bump up the bandwidth and see if anything changes. mg