Certain servers are not replying

[mon]

Addresses from 222.127.1.xxx return no data causing my time to mostly go out of sync. I am in Philippines and these are the addresses I usually get. I can ping these servers. It’s just the NTP that doesn’t work.

$ ntpdate -d arch.pool.ntp.org
12 Feb 19:48:55 ntpdate[290225]: ntpdate 4.2.8p17@1.4004-o Tue Jun  6 14:05:47 UTC 2023 (1)
Looking for host arch.pool.ntp.org and service ntp
222.127.1.19 reversed to mail.fortunetobacco.com
mail.fortunetobacco.com forward check lookup fail: Success
host found : 222.127.1.19 (mail.fortunetobacco.com)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
222.127.1.19: Server dropped: no data
222.127.1.20: Server dropped: no data
222.127.1.27: Server dropped: no data
222.127.1.18: Server dropped: no data

12 Feb 19:49:09 ntpdate[290225]: no server suitable for synchronization found

$ ntpdate -d ph.pool.ntp.org
12 Feb 19:49:44 ntpdate[290591]: ntpdate 4.2.8p17@1.4004-o Tue Jun  6 14:05:47 UTC 2023 (1)
Looking for host ph.pool.ntp.org and service ntp
222.127.1.19 reversed to mail.fortunetobacco.com
mail.fortunetobacco.com forward check lookup fail: Success
host found : 222.127.1.19 (mail.fortunetobacco.com)
transmit(222.127.1.19)
transmit(222.127.1.27)
transmit(45.249.226.5)
receive(45.249.226.5)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.27)
transmit(45.249.226.5)
receive(45.249.226.5)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.27)
transmit(45.249.226.5)
receive(45.249.226.5)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.27)
transmit(45.249.226.5)
receive(45.249.226.5)
transmit(222.127.1.18)
222.127.1.19: Server dropped: no data
222.127.1.27: Server dropped: no data
222.127.1.18: Server dropped: no data

server 45.249.226.5, port 123
stratum 2, precision -25, leap 00, trust 000
refid [61.239.100.17], root delay 0.047546, root dispersion 0.001221
reference time:      e97483cb.14416a4f  Mon, Feb 12 2024 19:47:23.079
originate timestamp: e974845f.1ecffed0  Mon, Feb 12 2024 19:49:51.120
transmit timestamp:  e974845f.102d1d78  Mon, Feb 12 2024 19:49:51.063
filter delay:  0.14577    0.14612    0.14755    0.14798
               ----       ----       ----       ----
filter offset: -0.002082  -0.002821  -0.003205  -0.004021
               ----       ----       ----       ----
delay 0.14577, dispersion 0.00087, offset -0.002082

12 Feb 19:49:53 ntpdate[290591]: adjust time server 45.249.226.5 offset -0.002082 sec

$ ntpdate -d pool.ntp.org
12 Feb 20:07:26 ntpdate[296003]: ntpdate 4.2.8p17@1.4004-o Tue Jun  6 14:05:47 UTC 2023 (1)
Looking for host pool.ntp.org and service ntp
host found : 222.127.1.21
transmit(222.127.1.21)
transmit(222.127.1.24)
transmit(222.127.1.26)
transmit(222.127.1.18)
transmit(222.127.1.21)
transmit(222.127.1.24)
transmit(222.127.1.26)
transmit(222.127.1.18)
transmit(222.127.1.21)
transmit(222.127.1.24)
transmit(222.127.1.26)
transmit(222.127.1.18)
transmit(222.127.1.21)
transmit(222.127.1.24)
transmit(222.127.1.26)
transmit(222.127.1.18)
222.127.1.21: Server dropped: no data
222.127.1.24: Server dropped: no data
222.127.1.26: Server dropped: no data
222.127.1.18: Server dropped: no data

12 Feb 20:07:35 ntpdate[296003]: no server suitable for synchronization found

Testing ph.pool.ntp.org on NTP Server Test results in these:

IPv4 test results
Result:OK
Server:222.127.1.26
Stratum:2
Offset:0.012259
Delay:0.31761

Result:OK
Server:222.127.1.19
Stratum:2
Offset:0.052838
Delay:0.29301

Result:OK
Server:222.127.1.21
Stratum:2
Offset:0.018172
Delay:0.33665

Result:OK
Server:222.127.1.24
Stratum:2
Offset:-0.014333
Delay:0.26657

[Badeand]

These 4 are the only problematic ones? Does other servers work?
Do you get different servers if you use asia.pool.ntp.org?
Do you have many NTP clients coming from the same IP address?

If you have IPv6, try 2.pool.ntp.org

[Kets_One]

I tried to ping 222.127.1.21 and 222.127.1.26.
Both seem non-responsive.

Am I correct in that the vendor-pool (arch.pool.ntp.org) you are trying to connect to resolves to the same ntp servers as your region (ph.pool.ntp.org)?
Is this correct behavior?

[Sebhoster]

Seems like someone recently joined a bunch of servers with consecutive IPs into the pool - all from the 222.127.1.* block.

To the original questions:

1. The three domain names you tested are somewhat redundant. ph.pool.ntp.org will only return servers that are part of the Phillipine zone. arch.pool.ntp.org is probably just an alias for pool.ntp.org. pool.ntp.org will give you the best results, but is weighted to return servers close to you - which means servers from the ph.pool.ntp.org zone.

2. All of the servers (IPs) in your logs are showing green in the monitoring, so they at least answer some requests.

3. Some more troubleshooting: Can you ping those servers? Can you do ntp queries manually against other servers?

It is strange that all those servers seem unresponsive to you but not to the monitoring endpoints… Is there maybe something on your network that is sending excessive ntp requests and gets your public IP ratelimited?

[apuls]

I don’t think they are in general unresponsive.
I’ve tested two IP earlier from a company server and they respond.

Also one test with NTP Check worked well.

Same server does not respond to my ntpdate -d command from germany and i have some packet loss on different nodes before.

Current Score for the IP is: 19.9 pool.ntp.org: Statistics for 222.127.1.21

[NTPman]

Hello @mon , you are not the only one noticing that there is some problem with the ntppool in the Philippines:

Problems connecting to servers in the Philippines (ph.pool.ntp.org)

Hello! We’re having problems using NTP Pool in the Philippines Both our and our vendor hostnames and the generic pool hostname resolve to an address in the range 203.177.21.121-4. Although the monitor scores for these servers are good, any NTP client I’ve used times-out when making a request. Any suggestions as to what might be going on gratefully received.

[jonathan.miles]

Yeah, we’re still having issues with Philippines NTP servers. Sometimes it works, but often not.

I believe the servers that were in the pool when I made the post quoted above have now been removed from the pool and been replaced with the 222.127.1.* servers.

It’s nice to know we’re not alone, but unfortunately I can’t provide any more insight.

[Bas]

To give more info:

bas@workstation:~$ nslookup ph.pool.ntp.org 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	ph.pool.ntp.org
Address: 222.127.1.23
Name:	ph.pool.ntp.org
Address: 222.127.1.27
Name:	ph.pool.ntp.org
Address: 222.127.1.21
Name:	ph.pool.ntp.org
Address: 222.127.1.20

bas@workstation:~$ nslookup 222.127.1.23
** server can't find 23.1.127.222.in-addr.arpa: NXDOMAIN

bas@workstation:~$ nslookup 222.127.1.27
** server can't find 27.1.127.222.in-addr.arpa: NXDOMAIN

bas@workstation:~$ nslookup 222.127.1.21
** server can't find 21.1.127.222.in-addr.arpa: NXDOMAIN

bas@workstation:~$ ntpdate -q -u 222.127.1.23
server 222.127.1.23, stratum 2, offset +0.024979, delay 0.25237
14 Feb 18:57:05 ntpdate[12895]: adjust time server 222.127.1.23 offset +0.024979 sec

They do run time…

And it seems the range if from Globe Telecoms, located in…euh the Philippines.

222.127.1.23 IP Address Details - IPInfo.io

City Peñaranda|
State Central Luzon

Hope this helps…but you should use pool.ntp.org not country specific.

Bas.

[mon]

@Badeand

These 4 are the only problematic ones? Does other servers work?

There are more problematic ones. They are all under 222.127.1.xxx. There are other servers that work but the IP address is not in that range. And they are not up most of the time.

@Badeand

Do you get different servers if you use asia.pool.ntp.org?

Yes and they all work fine.

@Badeand

Do you have many NTP clients coming from the same IP address?

I only have two machines configured to use *.pool.ntp.org. I am behind CGNAT so there may be more.

@Badeand

If you have IPv6, try 2.pool.ntp.org

I don’t have ipv6.

@Sebhoster

Some more troubleshooting: Can you ping those servers?

I can ping those servers.

@Sebhoster

Can you do ntp queries manually against other servers?

$ ntpdate -dq asia.pool.ntp.org
15 Feb 08:50:34 ntpdate[49834]: ntpdate 4.2.8p17@1.4004-o Tue Jun  6 14:05:47 UTC 2023 (1)
Looking for host asia.pool.ntp.org and service ntp
103.147.22.149 reversed to tw.ntp.twds.com.tw
host found : tw.ntp.twds.com.tw
transmit(103.147.22.149)
receive(103.147.22.149)
transmit(185.217.99.236)
transmit(46.19.96.19)
receive(185.217.99.236)
transmit(103.17.182.30)
receive(103.17.182.30)
receive(46.19.96.19)

server 103.147.22.149, port 123
stratum 2, precision -25, leap 00, trust 000
refid [218.73.139.35], root delay 0.003632, root dispersion 0.000488
reference time:      e977dd1e.aa8af43a  Thu, Feb 15 2024  8:45:18.666
originate timestamp: e977de5b.46b723a0  Thu, Feb 15 2024  8:50:35.276
transmit timestamp:  e977de5b.386d9ec3  Thu, Feb 15 2024  8:50:35.220
delay 0.15021, dispersion 0.00000, offset -0.006491

server 185.217.99.236, port 123
stratum 3, precision -24, leap 00, trust 000
refid [192.114.63.250], root delay 0.000580, root dispersion 0.017609
reference time:      e977ddfa.79083012  Thu, Feb 15 2024  8:48:58.472
originate timestamp: e977de5b.8853952e  Thu, Feb 15 2024  8:50:35.532
transmit timestamp:  e977de5b.6ba235e8  Thu, Feb 15 2024  8:50:35.420
delay 0.27658, dispersion 0.00000, offset -0.013424

server 46.19.96.19, port 123
stratum 2, precision -24, leap 00, trust 000
refid [194.58.207.20], root delay 0.046448, root dispersion 0.016449
reference time:      e977dc16.f9cb148f  Thu, Feb 15 2024  8:40:54.975
originate timestamp: e977de5b.c3415325  Thu, Feb 15 2024  8:50:35.762
transmit timestamp:  e977de5b.9ed4a22a  Thu, Feb 15 2024  8:50:35.620
delay 0.32527, dispersion 0.00000, offset -0.007555

server 103.17.182.30, port 123
stratum 2, precision -23, leap 00, trust 000
refid [133.243.238.243], root delay 0.106781, root dispersion 0.030624
reference time:      e977dcf0.b22dd029  Thu, Feb 15 2024  8:44:32.696
originate timestamp: e977de5b.da6ecc0a  Thu, Feb 15 2024  8:50:35.853
transmit timestamp:  e977de5b.d209a2b8  Thu, Feb 15 2024  8:50:35.820
delay 0.09067, dispersion 0.00000, offset +0.000182

15 Feb 08:50:35 ntpdate[49834]: adjust time server 103.17.182.30 offset +0.000182 sec

@Sebhoster

It is strange that all those servers seem unresponsive to you but not to the monitoring endpoints

Someone mentioned these servers could be blocking ntp requests from residential IP addresses.

@NTPman

Hello @mon , you are not the only one noticing that there is some problem with the ntppool in the Philippines:

I have not seen the IP address in that topic during my testing.

@Bas

but you should use pool.ntp.org not country specific.

I can’t.

@Sebhoster

• The three domain names you tested are somewhat redundant. ph.pool.ntp.org will only return servers that are part of the Phillipine zone. arch.pool.ntp.org is probably just an alias for pool.ntp.org. pool.ntp.org will give you the best results, but is weighted to return servers close to you - which means servers from the ph.pool.ntp.org zone.

[mon]

Can the server owners be contacted? Clarify what is really going on?

[alica]

Maybe just overloading in peak hours. My server in nearby TW is serving near 40x of its config bandwidth, and being knocked out every day during night peak…

[mon]

I’ve tested their servers in various times of the day and I’ve never seen them return anything.

[mon]

@apuls

Same server does not respond to my ntpdate -d command from germany

Is this from your home IP address?

[Sebhoster]

I tested the servers from my infrastructure (home and two cloud server from different providers, everything in Germany) and also did not get any response from these servers…

[Sebhoster]

I think the servers are overloaded - but all the time, not just during peaks.

The Phillipines is a country with a large population (>100,000,000) and a negligible amount of ntp pool servers. In other words, it is just a critically underserved zone.

I’ve spun up a VPS with chrony in the Phillipines to check, joined it into the pool, and it dropped out of the pool again after an hour, reaching over 20k requests per second with the default netspeed setting of 10Mb/s.

So the only long term solutions I see are to either get the pool zone refactoring @ask has been talking about for a while so clients in the Phillipines get more than those 10 servers, or to manually add servers from well served zones to the ph zone.

[apuls]

Sorry for the late reply.

Home ISP: no response
Work ISP: response
Server 4xDE, 1xNL, 1xLT, 1xMD, 4xUSA, 1xUK: no response

[mon]

@Sebhoster

So the only long term solutions I see are to either get the pool zone refactoring @ask has been talking about for a while so clients in the Phillipines get more than those 10 servers, or to manually add servers from well served zones to the ph zone.

I guess it remains broken until there is a fix? Is there a link to this discussion?

[Sebhoster]

Currently if I resolve arch.pool.ntp.org from an endpoint in the phillipines, i get different answers than just the ones you mentioned above. So the problem seems to be gone - for now?

The zone restructuring was discussed here:

Minor new features on the website

Yes, indeed. The new measurements of IPs served is preparing for building the GeoDNS configuration so that all countries are serviced by a minimum number of servers (20? 30?) and all the servers are (overall) getting a proportional amount of traffic as well as we can figure out. Indeed. “As close as possible” is more about minimizing the risk of asymmetric routing and dropped packets, not so much for accuracy. Years ago someone added the first server on some island in the south pacific. It h…

[giovane]

arch.pool.ntp.org is just an alias for pool.ntp.org.

So if your endpoint is in the Phillipines, then arch.pool.ntp.org is mapped to ph.pool.ntp.org

ph.pool.ntp.org now has 10 servers, so maybe that’s it. You can query dig ph.pool.ntp.org @e.ntpns.org. repetitively to get them.

What I find very dangerous is to have all NTP servers on the same /24 and same AS/provider

222.127.1.18
222.127.1.19
222.127.1.20
222.127.1.21
222.127.1.22
222.127.1.23
222.127.1.24
222.127.1.25
222.127.1.26
222.127.1.27

(more info on this: Deep Dive into NTP Pool's Popularity and Mapping | Proceedings of the ACM on Measurement and Analysis of Computing Systems)

[Sebhoster]

I did some more digging into the stats that my NTP server in the Phillipines produced.

Apparently my server, with Netspeed at not-zero-minimum (512k), and only occasionally available in the pool during the day due to large moves in the monitoring score, peaks at a NTP request rate higher than what I estimate for the entire country of Spain. In numbers: 120,000 requests per second.

At those peaks, 99% of the traffic is originating from just 10% of the client IPs, which correspondingly are heavily ratelimited.

So either the Philippines have a lot more devices per capita that query the pool than other countries and most of them behind some kind of NAT, or they have some seriously broken clients or setups.

Also, the number of requests my server got increased even during times where it was not active in the pool. The only thing I could think of leading to this is if somewhere the DNS responses from the pool are cached for far longer than they should?

This all sounds rather strange, so i think one solution might actually be to figure out why there are so many requests and not just to fight the symptom with more servers or a different distribution mechanism.

Do you have any ideas where to start? I was thinking of writing mails to the abuse teams of the IP addresses with the largest request rates, since those are only a couple dozen addresses from a handful of ISPs.