Server Unreachable from NTPPool

Server operators
#1

My server ntp.bollar.com was working fine with NTPPool until roughly a month ago, when I started getting NTP Pool: Problems with your NTP server (47.51.249.154) emails.

As far as I can tell, my server is setup correctly and has NTP traffic from pool clients before the error.
A test from ipvoid shows the server is up and accepting udp 123.

While researching, it appears there may not be a valid route from the test server to my server: https://trace.ntppool.org/traceroute/47.51.249.154

Traceroute to 47.51.249.154
 1 (139.178.64.41) AS54825  11.762  11.710
 2 (147.75.98.104) AS54825  5.479
 2 0.xe-0-0-17.dsr2.ewr1.packet.net (147.75.98.106) AS54825  15.810
 3 (198.16.4.214) AS54825  51.527
 3 0.ae3.bsr1.ewr1.packet.net (198.16.4.210) AS54825  0.683
 4 (192.80.8.11) AS54825  1.367
 4 0.et-0-0-7.bsr1.ewr2.packet.net (198.16.7.207) AS54825  1.532
 5 nyk-b2-link.telia.net (62.115.175.182) AS1299  1.008  0.991
 6  *  *
 7  *  *
 8  *  *
 9  *  *
10  *  *
11  *  *
12  *  *
13  *  *
14  *  *
15  *  *
16  *  *
17  *  *
18  *  *
19  *  *
20  *  *
21  *  *
22  *  *
23  *  *
24  *  *
25  *  *
26  *  *
27  *  *
28  *  *
29  *  *
30  *  *

Am I missing something? Is this something I can fix on my own?

Thanks.

1 Like
#2

Hi @bollar, have tried to get time from your server from here in the UK and from a few online tools and all report your server is not accessible. If you’re happy it is accessible from the internet then I would raise a ticket with your ISP. Sometimes they block ports “for security reasons” etc!

mtr --port 123 --udp ntp.bollar.com gives up…

  Host                                                                                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  4. 40ge1-3.core1.lon2.he.net                                                                                      0.0%    25   13.9  14.2  11.1  33.1   4.5
  5. 100ge13-2.core1.nyc4.he.net                                                                                    0.0%    25   81.0  79.5  78.0  83.1   1.4
  6. 100ge16-1.core1.ash1.he.net                                                                                    0.0%    25   83.0  85.1  82.0 102.6   5.3
  7. charter-net-hky-nc-as20115.100gigabitethernet6-1.core1.ash1.he.net                                             0.0%    25   84.5  85.2  83.8  91.5   1.6
  8. ???

mtr ntp.bollar.com gets a bit further…

  Host                                                                                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  4. 40ge1-3.core1.lon2.he.net                                                                                      0.0%     7   12.9  19.3  10.8  28.0   7.3
  5. 100ge13-2.core1.nyc4.he.net                                                                                    0.0%     7   80.7  80.5  78.0  90.5   4.5
  6. 100ge16-1.core1.ash1.he.net                                                                                    0.0%     7   96.6  87.9  81.7 102.3   9.1
  7. charter-net-hky-nc-as20115.100gigabitethernet6-1.core1.ash1.he.net                                             0.0%     6   83.8  83.9  83.5  84.3   0.3
  8. ???
  9. ???
 10. ???
 11. ???
 12. ???
 13. ???
 14. 047-051-249-154.biz.spectrum.com                                                                              16.7%     6  137.0 136.1 135.5 137.0   0.6
1 Like
#3

Same here: no response on udp/123 from several places in UK and US. Suspect firewalling by your ISP.

#4

I get slightly different results:

root@oguard:~# mtr -bw -c2 -r ntp.bollar.com
Start: 2020-07-29T14:34:05+1000
HOST: oguard                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                                                   100.0     2    0.0   0.0   0.0   0.0   0.0
  2.|-- bri-pow-que-crt3-be-100.tpg.com.au (60.240.241.1)                      0.0%     2    5.9   6.4   5.9   6.9   0.8
  3.|-- syd-apt-ros-crt1-be-50.tpg.com.au (203.219.107.74)                     0.0%     2   26.0  24.0  22.1  26.0   2.8
  4.|-- syd-apt-ros-int1-eth8-3.tpgi.com.au (203.29.134.67)                    0.0%     2   24.4  23.2  21.9  24.4   1.8
  5.|-- 100ge13-1.core1.sjc1.he.net (216.218.139.233)                          0.0%     2  174.1 172.9 171.6 174.1   1.8
  6.|-- 10ge7-2.core1.sjc2.he.net (72.52.92.118)                               0.0%     2  172.3 171.9 171.5 172.3   0.5
  7.|-- e0-36.core2.sjc2.he.net (184.104.192.214)                              0.0%     2  174.5 173.7 172.8 174.5   1.2
  8.|-- charter-20115-as20115.port-channel2.core2.sjc2.he.net (64.62.153.206)  0.0%     2  173.2 172.9 172.5 173.2   0.5
  9.|-- bbr02snjsca-bue-1.snjs.ca.charter.com (96.34.3.108)                    0.0%     2  178.7 175.7 172.7 178.7   4.3
 10.|-- bbr01rvsdca-bue-8.rvsd.ca.charter.com (96.34.0.177)                    0.0%     2  186.5 187.6 186.5 188.6   1.5
 11.|-- bbr01dllstx-bue-6.dlls.tx.charter.com (96.34.0.20)                     0.0%     2  219.5 217.4 215.3 219.5   3.0
 12.|-- crr01ftwotx-bue-2.ftwo.tx.charter.com (96.34.2.33)                     0.0%     2  221.1 219.7 218.2 221.1   2.1
 13.|-- dtr01kllrtx-bue-3.kllr.tx.charter.com (96.34.112.181)                  0.0%     2  216.8 218.4 216.8 219.9   2.2
 14.|-- ???                                                                   100.0     2    0.0   0.0   0.0   0.0   0.0
 15.|-- 047-051-249-154.biz.spectrum.com (47.51.249.154)                      50.0%     2  233.5 233.5 233.5 233.5   0.0
root@oguard:~# mtr -bw -c2 -r --port 123 --udp ntp.bollar.com
Start: 2020-07-29T14:48:51+1000
HOST: oguard                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                                                   100.0     2    0.0   0.0   0.0   0.0   0.0
  2.|-- bri-pow-que-crt3-be-100.tpg.com.au (60.240.241.1)                      0.0%     2    6.4   6.1   5.8   6.4   0.4
  3.|-- syd-apt-ros-crt1-be-50.tpg.com.au (203.219.107.74)                     0.0%     2   22.0  22.1  22.0  22.2   0.1
  4.|-- syd-apt-ros-int1-eth8-3.tpgi.com.au (203.29.134.67)                    0.0%     2   23.3  23.1  23.0  23.3   0.2
  5.|-- 100ge13-1.core1.sjc1.he.net (216.218.139.233)                          0.0%     2  172.5 176.4 172.5 180.2   5.5
  6.|-- 100ge1-1.core1.sjc2.he.net (184.105.65.114)                            0.0%     2  172.0 172.3 172.0 172.6   0.4
  7.|-- e0-36.core2.sjc2.he.net (184.104.192.214)                              0.0%     2  173.2 172.5 171.7 173.2   1.1
  8.|-- charter-20115-as20115.port-channel2.core2.sjc2.he.net (64.62.153.206)  0.0%     2  188.7 180.9 173.1 188.7  11.1
  9.|-- bbr02snjsca-bue-1.snjs.ca.charter.com (96.34.3.108)                    0.0%     2  175.1 176.8 175.1 178.5   2.5
 10.|-- bbr01rvsdca-bue-8.rvsd.ca.charter.com (96.34.0.177)                    0.0%     2  186.7 187.3 186.7 187.9   0.8
 11.|-- bbr01dllstx-bue-6.dlls.tx.charter.com (96.34.0.20)                     0.0%     2  218.4 219.4 218.4 220.3   1.4
 12.|-- crr01ftwotx-bue-2.ftwo.tx.charter.com (96.34.2.33)                     0.0%     2  223.8 219.5 215.2 223.8   6.1
 13.|-- dtr01kllrtx-bue-3.kllr.tx.charter.com (96.34.112.181)                  0.0%     2  220.0 219.7 219.3 220.0   0.5
 14.|-- ???                                                                   100.0     2    0.0   0.0   0.0   0.0   0.0

I’m not particularly surprised that the last mtr hop doesn’t work when using UDP, because the last hop will be the host itself, and mtr will not be composing a correct NTP packet, so the NTP server should not reply.

I ran up this server as a source on my machine (with burst iburst noselect enabled), and at one point I got enough responses that it could calculate jitter:

 47.51.249.154   .PPS.            1 u   34   64    7  230.999    1.714   0.618

But now it has gone back to being unresponsive.

My guess would be that rather than this being a routing problem on the ISP side, the volume of requests going to this server is high enough that it filled connection tracking tables in the host or an upstream firewall.

1 Like
#5

Thanks to @elljay and @paulgear for your diagnostic efforts.

It does seem that there is commonality with the failures starting inside the Charter Spectrum network. This is a business account that isn’t supposed to have port blocking, so there is a possibility of misconfiguration in their network. I’ll create a trouble ticket and see.

I also won’t completely discount the possibility of the server being overloaded and I will look into that. It has about 600 clients now, which is well below the peak of over 13K a month ago.

If there are some other thoughts, please let me know.

#6

To be clear, I was not suggesting that it was load on the server, but a full connection-tracking table somewhere, which is often artificially low (at least, the default on Linux is). If you can give some details about the host (OS, NTP server, firewall configuration, etc.), we may be able to narrow this down.

Usually it’s best to exclude NTP from connection tracking if that’s possible in your setup.

2 Likes