Problem with IPv4 servers

Server operators
1

Hi,

About a week ago, the IPv4 part of the servers stoped working for the pool. But they do work, I know this because I you them via IPv4 for hundreds of devices here. Also IPv6 still works fine. I thought it may have been a firewall issue, but dont find any blocks that would affect it, plus they appear reachable by ntppool:

Traceroute to 200.59.16.50
 1 (139.178.64.41) AS54825  47.081  47.038

…
16 (186.176.7.74) AS20299 71.028 70.977
17 ntp3.fratec.net (200.59.16.50) AS262149 82.264 83.180

another one:

Traceroute to 200.59.21.5
1 (139.178.64.41) AS54825 47.131 47.077
…
16 (200.59.17.43) AS262149 73.008
16 www.fratec.net (200.59.21.5) AS262149 81.316

Still they all give timeout to Newark Monitoring Station:

ts_epoch,ts,offset,step,score,leap,error
1611337815,“2021-01-22 17:50:15”,0,-5,-100,“i/o timeout”
1611336733,“2021-01-22 17:32:13”,0,-5,-100,“i/o timeout”
1611335662,“2021-01-22 17:14:22”,0,-5,-100,“i/o timeout”
1611334581,“2021-01-22 16:56:21”,0,-5,-100,“i/o timeout”
1611333499,“2021-01-22 16:38:19”,0,-5,-100,“i/o timeout”
1611332422,“2021-01-22 16:20:22”,0,-5,-100,“i/o timeout”
1611331355,“2021-01-22 16:02:35”,0,-5,-100,“i/o timeout”
1611330291,“2021-01-22 15:44:51”,0,-5,-100,“i/o timeout”
1611329218,“2021-01-22 15:26:58”,0,-5,-100,“i/o timeout”
1611328121,“2021-01-22 15:08:41”,0,-5,-100,“i/o timeout”

200.59.16.50

Puntuación actual: -100.0 (sólo los servidores con una puntuación mayor que 10 se usan en el equipo)

History

12Thu 2112Fri 2212200-20-50-100Offset monitoring and scores for 200.59.16.50

Monitoring Station:Newark, NJ, US (-100)

What can I do to find the problem here? Whats Newarks IP so I can monitor it?

2

Newark is 139.178.64.42 and hasn’t seen any NTP responses from 200.59.16.50 since 2021-01-13 19:52 UTC

ping and default traceroute from Newark reach the NTP server, but NTP isn’t getting through.

However when I try traceroute from Newark to the NTP server using destination UDP port 123 I see a routing loop.

traceroute -p 123 200.59.16.50
traceroute to 200.59.16.50 (200.59.16.50), 30 hops max, 60 byte packets
1 gateway (139.178.64.41) 16.146 ms 16.104 ms 16.068 ms
2 * * *
3 0.ae3.bsr2.ewr1.packet.net (198.16.4.214) 0.898 ms 0.888 ms 0.ae2.bsr2.ewr1.packet.net (198.16.4.212) 0.868 ms
4 0.et-0-0-7.bsr1.ewr2.packet.net (198.16.7.207) 7.634 ms 7.612 ms 7.589 ms
5 nyk-b2-link.telia.net (62.115.175.182) 0.980 ms 0.959 ms 0.916 ms
6 nyk-bb3-link.ip.twelve99.net (213.155.130.29) 33.017 ms 33.031 ms 33.267 ms
7 rest-bb1-link.ip.twelve99.net (62.115.141.244) 8.424 ms 8.652 ms 8.633 ms
8 boca-b2-link.ip.twelve99.net (62.115.123.29) 32.725 ms 33.140 ms 32.860 ms
9 asurnet-svc071312-ic357662.c.telia.net (62.115.40.211) 33.441 ms 32.936 ms boca-b2-link.ip.twelve99.net (62.115.118.162) 34.795 ms
10 asurnet-svc071312-ic357662.c.telia.net (62.115.40.211) 34.403 ms ae0.nmi-mx2020-2.north-miami.fl.usa.cwc.com (69.79.102.1) 34.251 ms 36.951 ms
11 69.79.106.49 (69.79.106.49) 56.646 ms 56.676 ms 56.914 ms
12 * * *
13 * * *
14 * * *
15 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 71.076 ms 71.397 ms 76.491 ms
16 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 81.921 ms 86.875 ms 86.891 ms
17 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 71.280 ms 69.935 ms 71.293 ms
18 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 87.502 ms 80.816 ms 82.226 ms
19 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 71.312 ms 71.922 ms 71.448 ms
20 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 87.060 ms 80.977 ms 81.961 ms
21 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 71.446 ms 71.997 ms 71.368 ms
22 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 82.522 ms 81.859 ms 80.975 ms
23 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 72.252 ms 76.857 ms 71.455 ms
24 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 83.103 ms 82.473 ms 82.673 ms
25 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 70.460 ms 76.790 ms 71.122 ms
26 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 81.270 ms 82.499 ms 82.464 ms
27 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 71.698 ms ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 83.564 ms sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 76.740 ms
28 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 83.490 ms 84.334 ms 83.422 ms
29 sfp-sfpplus1.mikrotik-bb-router-gnd1.fratec.net (200.59.17.43) 72.663 ms 70.875 ms 77.155 ms
30 ether1.mikrotik-bb-desa1.fratec.net (200.59.19.50) 87.484 ms 87.406 ms 83.805 ms

3

I just rebooted one of the routers so that may have coused the loop, but I can reach Newark from that server just fine:

root@stratum1:~# traceroute 139.178.64.42
traceroute to 139.178.64.42 (139.178.64.42), 30 hops max, 60 byte packets
1 ether3.mikrotik-bb-zeus.fratec.net (200.59.16.33) 0.193 ms 0.152 ms 0.156 ms
2 186.176.7.73 (186.176.7.73) 0.745 ms 0.793 ms 0.766 ms
3 186.32.0.217 (186.32.0.217) 1.012 ms 1.007 ms 1.018 ms
4 * 190.106.192.237 (190.106.192.237) 52.398 ms *
5 mai-b1-link.telia.net (62.115.56.164) 40.884 ms 42.800 ms 51.117 ms
6 ash-bb2-link.ip.twelve99.net (62.115.120.176) 82.645 ms 72.525 ms rest-bb1-link.ip.twelve99.net (62.115.119.230) 68.002 ms
7 * nyk-bb3-link.ip.twelve99.net (62.115.141.245) 76.663 ms *
8 nyk-b2-link.ip.twelve99.net (62.115.137.99) 71.674 ms 71.693 ms nyk-b2-link.ip.twelve99.net (213.155.130.28) 71.854 ms
9 packethost-ic-345229-nyk-b2.c.telia.net (62.115.175.183) 83.129 ms 82.109 ms 71.260 ms
10 0.et-0-0-7.bsr2.ewr1.packet.net (192.80.8.10) 71.881 ms 82.375 ms 0.et-0-0-1.bsr1.ewr1.packet.net (198.16.7.206) 74.278 ms
11 0.ae2.dsr1.ewr1.packet.net (198.16.4.213) 90.639 ms 0.ae2.dsr2.ewr1.packet.net (198.16.4.215) 84.180 ms 95.069 ms
12 147.75.98.105 (147.75.98.105) 103.039 ms 87.503 ms 86.315 ms
13 monewr1.ntppool.net (139.178.64.42) 77.766 ms 78.827 ms 88.124 ms

4

Rebooting the router did give a clue:

ts_epoch,ts,offset,step,score,leap,error
1611342156,“2021-01-22 19:02:36”,0,-5,-94.3,“i/o timeout”
1611341063,“2021-01-22 18:44:23”,-0.003019731,1,-94,0,
1611340001,“2021-01-22 18:26:41”,0,-5,-100,“i/o timeout”

while that link was down Newark did get an answer. Could it be that Newark needs the NTP packet to enter / exit from the same provider on my end? I’ll do some more testing

5

From 18:41 - 18:48 I saw NTP responses getting through.

Now I’m seeing repeatable NTP blockage. I’ll characterize it over the next couple of hours.

6

Just in case I tested from other networks and it works fine, for example:

https://keetweej.vanheusden.com/query_ntp_do2.php?host=ntp3.fratec.net

7

There is a hard block for port 123 at “COLUMBUS-NETWORKS”.
Are you able to change routing from your side?

Use ntp port, 123

$ traceroute -n -U -p 123 200.59.16.50
traceroute to 200.59.16.50 (200.59.16.50), 30 hops max, 60 byte packets
1 139.178.64.41 21.376 ms 21.321 ms 21.292 ms
2 * * *
3 198.16.4.208 2.413 ms 198.16.4.212 0.785 ms 198.16.4.214 0.771 ms
4 198.16.7.207 1.546 ms 192.80.8.11 1.791 ms 198.16.7.207 1.761 ms
5 62.115.175.182 0.953 ms 0.918 ms 1.149 ms
6 213.155.130.29 33.071 ms 32.921 ms 33.131 ms
7 62.115.141.244 8.850 ms 8.394 ms 9.032 ms
8 62.115.123.29 32.779 ms 32.724 ms 33.179 ms
9 62.115.40.211 33.081 ms 33.130 ms 32.946 ms
10 69.79.102.1 34.182 ms 33.978 ms 33.953 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
etc.

Use a non-NTP port, 124

$ traceroute -n -U -p 124 200.59.16.50
traceroute to 200.59.16.50 (200.59.16.50), 30 hops max, 60 byte packets
1 139.178.64.41 14.005 ms 13.962 ms 13.925 ms
2 * * *
3 198.16.4.210 0.772 ms 198.16.4.214 0.966 ms 1.192 ms
4 192.80.8.11 1.322 ms 198.16.7.207 1.379 ms 1.604 ms
5 62.115.175.182 1.072 ms 1.295 ms 1.252 ms
6 213.155.130.29 32.912 ms 33.096 ms 32.913 ms
7 62.115.141.244 8.853 ms 9.124 ms 8.230 ms
8 62.115.123.29 33.177 ms 32.900 ms 33.112 ms
9 62.115.40.211 32.983 ms 33.465 ms 33.058 ms
10 69.79.102.1 34.363 ms 34.151 ms 34.108 ms
11 69.79.106.49 57.177 ms 56.620 ms 69.79.102.1 35.647 ms
12 * * *
13 * * *
14 * * *
15 186.179.72.111 89.845 ms 89.896 ms 89.618 ms
16 200.59.17.41 90.921 ms 85.760 ms 90.876 ms
17 200.59.16.50 87.105 ms 87.076 ms 81.892 ms

Look at ownership:
54825 | 198.16.4.210 | 198.16.4.0/22 | US | arin | 2012-09-05 | PACKET, US
54825 | 192.80.8.11 | 192.80.8.0/24 | US | arin | 2014-12-17 | PACKET, US
1299 | 62.115.175.182 | 62.115.0.0/16 | SE | ripencc | 2001-08-09 | TELIANET Telia Carrier, SE
1299 | 213.155.130.29 | 213.155.128.0/19 | SE | ripencc | 2000-02-18 | TELIANET Telia Carrier, SE
1299 | 62.115.141.244 | 62.115.0.0/16 | SE | ripencc | 2001-08-09 | TELIANET Telia Carrier, SE
1299 | 62.115.123.29 | 62.115.0.0/16 | SE | ripencc | 2001-08-09 | TELIANET Telia Carrier, SE
1299 | 62.115.40.211 | 62.115.0.0/16 | SE | ripencc | 2001-08-09 | TELIANET Telia Carrier, SE
23520 | 69.79.102.1 | 69.79.102.0/24 | US | arin | 2003-09-24 | COLUMBUS-NETWORKS, US
23520 | 69.79.106.49 | 69.79.106.0/24 | US | arin | 2003-09-24 | COLUMBUS-NETWORKS, US