Aren’t most people behind a CGNAT these days? We used to have a public IP address 10 years ago. Now it has a premium price.
I don’t know any better than you do in this regard. I appreciate you helping out on this issue though.
Doesn’t seem to be the case where I live, in Norway. But maybe it’s more common in Asia than Europe?
I looked at a 60-second pcap from Sebhoster’s server. Typical caveats: only examined one server for a short time, traffic may vary over time, etc.
Preliminary analysis below. The behavior is unlike other high-request rate systems that I’ve looked at.
NTP requests were sent by 17,564 clients during the 60-seconds. 99.6% of the requests were sent by 152 clients! Most clients sent requests at a steady rate of 300-1500 during the entire period.
This is probably not a NAT/VM related situation.
Except for the absolute rate, all 152 clients exhibit similar behavior: UDP source port is constant, the requests show alarm=0, version=4, stratum=0. The NTP request transmit time (set by the client when it sends the request) are all in error by either 20.5, 41, or 82 years.
Looking again at the NTP request transmit time stamp the fractional part is always <= 0.232 seconds. A similar behavior was seen in recent years due to a bug in systemd’s timesyncd service. The Philippines situation differs in other aspects.
My guess is buggy client software is to blame.
2 Likes
Just block these clients?
That only solves the symptom, not the problem. And it would need to be done by every operator of any pool-joined server in the Phillipines.
I think I’ll start writing emails to the providers responsible for the IP addresses and see if that helps
Hi o/
Did you have any reply to your emails?
We’re still having the same problem:
# ntpdate -d 0.rhel.pool.ntp.org
mail.fortunetobacco.com forward check lookup fail: Success
host found : 222.127.1.19 (mail.fortunetobacco.com)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
transmit(222.127.1.19)
transmit(222.127.1.20)
transmit(222.127.1.27)
transmit(222.127.1.18)
222.127.1.19: Server dropped: no data
222.127.1.20: Server dropped: no data
222.127.1.27: Server dropped: no data
222.127.1.18: Server dropped: no data
Can these servers be removed from the pool?
Thanks in advance.
I ran mtr (traceroute) from New York City for two of these server addresses: 122.127.1.20 and 122.127.1.22. Both addresses showed the same behavior:
Target port 123 (NTP) No packets reach server
Target port 122 or 124 At least 80% of the packets reach NTP server
Ports 122 and 124 are used for comparison.
It looks like AS4775 is intentionally blocking NTP messages from New York.
I checked NTP connectivity from another client located in suburban Chicago and saw zero loss (NTP connectivity was 100%)
Last night from ~200 vantage points in the Philippines and beyond towards 122.127.1.26: not a single response.
At the same time, the score was 20. But it is interesting to see that multiple monitors have score -100.
Looks indeed like there is heavy selective filtering.
New measurement with the same vantage points, and some more, probing 122.127.1.26 once an hour for the next 24 hours to see whether there is any variation throughout the day.
Works fine here:
root@workstation:~# ntpdate -q 222.127.1.26
server 222.127.1.26, stratum 2, offset +0.004966, delay 0.34708
21 May 18:06:19 ntpdate[20110]: adjust time server 222.127.1.26 offset +0.004966 sec
- ~335 probes/vantage points in the Philippines, Asia, and beyond
- 23520 requests overall
- no response whatsoever
I know the system worked in the past, but seeing no response whatsoever got me wondering. But it does work.
Maybe networks are blocked because of bigh number of requests?
My server does this if an IP(gateway) is polling too much.
It simply stops responding.
I see ISP’s request massive over 1 IP to my server, they are out of luck and block themselves.
Nice work. Two different magic number fingerprints should be a big help in tracking down the problematic firmware or software.
Can you help me find more about the <= 0.232 bug in timesyncd? My admittedly poor web search skills have failed me so far.
It was a feature. timesyncd put nanoseconds directly in the fractional part of the timestamp. It was changed recently in this commit:
This means it’s now more difficult to identify the timesyncd client when it it’s flooding its server due to this bug:
1 Like
Thank you, Miroslav! You are a priceless resource to the community.
I still wonder about 20.5 years and its multiples. It’s probably something to do with whatever else is setting the clock on the systems plaguing the Philippines pool (and who knows which other parts of the pool are affected but not so severely).
1 Like
Hello, everyone,
We are still encountering this issue in the Philippines. What possible actions can we take to address it?
In the Philippines, is it necessary for us to temporarily abandon the use of pool.ntp.org and rely on other NTP servers such as time.aws.com?
NTP Service by local authority should be usable.
https://bagong.pagasa.dost.gov.ph/astronomy#philippine-standard-time
@alica
Thank you!
“ntp.pagasa.dost.gov.ph” works.
But we are happy if we can continue using “pool.ntp.org” in the Philippines.
We don’t use “ph.pool.ntp.org” but rather “pool.ntp.org”. However, we always get 4 addresses ranging from “222.127.1.18 to 222.127.1.27”.
If even one of these 4 addresses could be from another source, such as “asia.pool.ntp.org”, our problem would be resolved.
Is implementing this countermeasure difficult for you?
Try asia.pool.ntp.org. The global pool @ pool.ntp.org will try to detect your location via GeoIP, either based on your IP address, or, lacking that, based on your resolver’s IP address, and will then only provide you with servers from the zone it thinks you are in. (In a way, the term “global” seems misleading, because one does not get servers from all over the world.)
As per some testing I did previously, asia.pool.ntp.org should also return addresses from Asia beyond a single country zone, in a few cases even from outside Asia (just as this can happen with a country zone in case a server’s location was misdetected at registration time, or, arguably more frequently, because it was manually added to the Asia zone, or some country zone considered to be in Asia, to add capacity to the respective zone).
Or you pick one or more explicit country zones apart from your own which have a sufficient number of stable servers in them, and which are “close” enough to your location (with “close” put in quotes because it is not primarily about geographic or network topology proximity, but about providing good time service).