Monitoring station unable to query NTP servers

Server operators
1

For years now I’ve had a problem that for some reason my NTP servers work fine via one Internet provider but not the other one. So my guess is that somewhere between the monitoring station and my servers port 123 is being blocked. Right now I set up 2 servers for testing, one works the other does not, both can be reached by the monitoring station just fine:

provider 1 works:

Traceroute to 200.59.21.5
 1  *  *
 2 (169.254.74.0)  0.132  0.159
 3 (10.253.4.207)  0.121
 3 (10.253.4.209)  0.212
 4 (10.253.4.147)  9.348
 4 (10.253.4.149)  1.052
 5 (62.115.179.210) AS1299  0.439
 5 nyk-b4-link.ip.twelve99.net (62.115.180.4) AS1299  0.441
 6 nyk-bb1-link.ip.twelve99.net (62.115.114.178) AS1299  1.253  1.133
 7 (62.115.136.201) AS1299  7.036
 7 rest-bb1-link.ip.twelve99.net (62.115.141.244) AS1299  7.790
 8 mai-b2-link.ip.twelve99.net (62.115.120.177) AS1299  31.769  31.737
 9 navega-svc070684-ic356808.ip.twelve99-cust.net (62.115.56.165) AS1299  32.312  31.536
10  *  *
11  *  *
12 (186.176.7.74) AS262197  81.198  81.331
13 www.fratec.net (200.59.21.5) AS262149  81.141  71.528

provider 2 does not work:

Traceroute to 200.59.20.230
 1  *  *
 2 (169.254.74.0)  0.171
 2 (169.254.74.1)  0.125
 3 (10.253.4.205)  0.107
 3 (10.253.4.207)  0.084
 4 (10.253.4.149)  1.222
 4 (10.253.4.147)  0.669
 5 (62.115.179.210) AS1299  0.344
 5 nyk-b4-link.ip.twelve99.net (62.115.180.4) AS1299  0.349
 6 nyk-bb1-link.ip.twelve99.net (62.115.114.178) AS1299  1.317  1.334
 7 rest-bb1-link.ip.twelve99.net (62.115.141.244) AS1299  8.742  7.133
 8 boca-b2-link.ip.twelve99.net (62.115.123.29) AS1299  32.434  32.461
 9 (62.115.11.225) AS1299  32.261
 9 asurnet-svc080563-ic370378.ip.twelve99-cust.net (62.115.149.31) AS1299  32.474
10 (69.79.102.1) AS23520  32.700
10 ae2.brx-mx2020-2.boca-raton.fl.usa.cwc.com (69.79.100.5) AS23520  32.448
11 (69.79.102.1) AS23520  32.467
11 (69.79.106.49) AS23520  52.216
12 (69.79.106.49) AS23520  52.502  52.001
13  *  *
14 111-72-179-186.ufinet.co.cr (186.179.72.111) AS52468  69.117  68.831
15 (200.59.20.230) AS262149  69.357

Now using wetwiz.net both look ok:

NTP Server Test for 200.59.21.5

NTP Server Test for 200.59.20.230

But if I try to add the second one to the pool I get

200.59.20.230

Could not check NTP status

So, I need help sorting this out, I need to pinpoint the problem so whoever is responsible can fix it. If anyone has an idea how to find the problem, please let me now. Thanks

2

I checked NTP reachability from a couple of my clients. NTP worked for the client hosted on Comcast, but did not for the client hosted on AT&T. I checked traceroute (mtr).

It is important to probe using multiple UDP destination ports, in particular NTP (UDP port 123) and some other UDP port.

NOTE: mtr and traceroute options may vary.
I used:
mtr -n --udp -P 123 200.59.20.230
and
mtr -n --udp -P 124 200.59.20.230
The results showed blockage, but some nodes did not respond to traceroute. (I sent the details to Profile - jfrater - NTP Pool Project).

Next I logged onto monsjc2, the NTP Pool monitor and ran the same commands and also emailed the full traceroute to jfrater . To highlight the difference.

Port 124
Loss% Snt Last Avg Best Wrst StDev
7. 64.86.160.12 61.1% 18 1.3 1.6 1.2 2.2 0.4
8. 193.251.143.55 50.0% 18 1.6 1.7 1.3 2.2 0.3
9. 193.251.242.2 0.0% 18 74.6 42.3 1.4 75.0 37.2
10. 193.251.242.2 0.0% 18 74.6 74.8 74.4 75.5 0.2
11. 193.251.254.124 0.0% 18 74.9 101.6 74.9 123.6 24.2
12. 200.59.20.230 0.0% 18 122.3 122.5 121.7 124.1 0.7 <<< NTP server

Port 123

  1. 64.86.160.3 30.0% 10 1.7 1.7 1.2 2.3 0.4
  2. 64.86.160.3 10.0% 10 1.8 3.3 1.2 15.3 4.5
  3. 193.251.242.2 0.0% 10 74.5 45.7 1.4 75.2 37.7
  4. 193.251.242.2 66.7% 10 75.1 74.7 74.5 75.1 0.3
    11 no response

This appears to be an NTP specific block close to 193.251.254.124 (Orange - OINIS)

3

Occasionally the Port 123 trace shows
1220 2023-02-26 01:42:12.761014 193.251.254.124 → 147.75.202.162 ICMP 110 Destination unreachable (Communication administratively filtered)

This suggests the blockage is intentional.

4

TLDR: AT&T blocks packets coming from its network whose source port is 123 on IPv4.

2 Likes
5

AT&T doesn’t always block 123 it seems to happen when “congestion” rules are put into place. Overnight I can have a perfect 20 then at 5:50 am the monitor can’t reach me. It varies over the day and also varies over the weekend. I also see the congestion rules come into play where they have blocked 5xxx port ranges and I cannot access my NAS services from the public internet.

1 Like