The following NTP servers in the Philippines don’t work, even though their scores are higher than 10.
222.127.1.19 ~ 27
When we resolve “pool.ntp.org,” the resolved address is one of the addresses above, so we can’t use “pool.ntp.org” now.
This issue is similar to the one discussed in this thread:
However, the IP addresses are different, so I believe this is a separate problem and created a new topic. I’m sorry if I should have used the old thread.
From here, the details of the issue:
Using ntpdate, these servers don’t respond even with a 10-second timeout.
$ ntpdate -qv -t10 222.127.1.19
28 Jun 12:17:43 ntpdate[30485]: ntpdate 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1)
28 Jun 12:17:53 ntpdate[30485]: no server suitable for synchronization found
This site reported these servers as “Error; Timeout.”
On the other hand, these sites report those servers as OK:
It appears that these servers work for only a limited number of clients, not for everyone.
I think these servers should be deleted soon, otherwise people in the Philippines can’t use pool.ntp.org.
I ran a series of traceroutes and suspect there is intentional blockage by AS4775. The blockage is specific to NTP (UDP port 123) and depends on the NTP request source address.
"In the Philippines, we consistently receive four addresses ranging from “222.127.1.18 to 222.127.1.27” for NTP servers. The current situation indicates that servers with the same administrator, configuration, and location are being selected. As a result, access restrictions imposed by server or network administrators, whether intentional or unintentional, are impacting all servers.
Is it not possible to implement a method that allows for the inclusion of NTP servers from different environments? For instance, could we consider selecting one or two addresses from a higher-level domain, such as “asia.pool.ntp.org,” among the four addresses returned by DNS?"
That is my suggestion also. Addresses from 222.127.1.18 to 222.127.1.27 are all known to be advertised by AS4775, GLOBE-TELECOM-AS Globe Telecoms, PH. This can be quickly and easily programmatically determined.
The pool DNS server could decide to ensure that IPs from at least two or three different ASNs are returned, by expanding the scope of where it will draw IPs from (only) if necessary.
Returning four IPs from a single ASN risks a shared bad fate for all of them.
Based on my understanding, each server.Loc has an API called GetASN(), and comparing ASNs should not be difficult. Another idea is to use distance. If all distances are almost the same, it might be safer to select another server.
Has there been any progress on this? We have customers using devices that are shipped with our vendor domain in the Philippines who are reporting that the devices are unable to set the correct time. Our investigations show very similar symptoms to those described here.
All of the threads relating to this issue have been quiet for the past year. I’m wondering if the problem went away and has recently resurfaced, or has it just been bad consistently?
Did you happen to see my recent post List of trackers - #5 by gombadi where I described the problems I had setting up a server in Manila?
lt is a difficult location to add a server because of the load issues the server experiences. I would like to be able to add a server there but until I find a solution to the load issue I can’t see it happening.
When even the currently lowest “official” setting results in traffic peaks of sometimes double-digit Mbit/s or even higher in some zones, that prevents many if not most smaller servers that provide much of the capacity in many of the better-served zones from joining.
There is a way to set lower netspeed values even today (and also other values outside the strict granularity of what the management page offers). But it requires some fiddling with low-level details under the hood of the management page that tech-savvy enthusiastic people may use, but that is too cumbersome to reach the scale needed to make a difference in underserved zones. I.e., people who’d be happy to join the pool but don’t have the know-how/time/resources/inclination to dig into such low-level aspects, sometimes repeatedly until a satisfactory setting is found, will not be able to join. Which is a pity, because looking through this forum, this comes up time and time again. But without resolution leaves potential server hosts frustrated and turning away from the pool, when, as you rightly write, “[e]very single server counts”.
Sure, this wouldn’t solve the issue all by itself, and some “cover” is needed by bigger servers to allow smaller ones to thrive in their shadow - not only when a “small” server would be the only one in its country zone. But it would be a starting point.
An argument against small netspeed values often seen is that the resulting allocation of load would not be “accurate”. But not sure how relevant that is in an underserved zone where people would be happy to be able to join the pool at all, regardless of whether the load they get is an “accurate” share of the load in that zone. Or even in better-served zones.
I am not even sure what the point of that “accuracy” is. What I (in mostly better-served zones), and I guess other people, especially those unable to join the pool in underserved zones, are concerned about is the absolute load they get, because that is what determines whether it is ok, or too much.
Whether that load now accurately reflects the share of the overall load of that zone, i.e., a server’s relative netspeed in relation to the sum of netspeeds of all active servers in that zone (as shown in the “Client distribution” section of the management page of each active server) is definitely interesting, and certainly gives various indications as to a server’s performance, or about the zone it is in, and the “health” of both.
But the accuracy of that relative share seems pretty much secondary to the question of whether a server can cope with the absolute traffic rate a certain setting induces. Case in point being that even the pool pages say it is only a relative value, and the fact that the same netspeed setting results in wildly differing absolute traffic loads depending on the zone being looked at.
I think a likely problem may be that there is a minimum speed value below which it’s no longer possible to limit how long clients hold on to the IP they got handed, and by extension not possible to regulate the lower bound of load on any given server in an underserved zone no matter how low its speed is set to.
I have previously suggested that zones should have a minimum number of available servers, and if the number of available servers falls below that value then extra servers could be “drafted in” from geographically adjacent zones or even globally, on the basis that high RTT on some peers is better than no service at all. I think this might make small zones with high load viable without having to do constant plea drives for resources.
On the other hand if it can be shown that adding even lower settings of net speed is still effective at controlling server load then of course that is an easier option. Though again I would worry that it’s asking a lot of server admins to actually be able to predict what their net speed should be set to, given that there is no actual correlation between stated bandwidth and actual load (which will be more about packets per second). The process would continue to be, “if your Internet connection falls over, keep reducing net speed until it doesn’t,” just with a bit more chance that the lowest selectable speed would actually be viable.
I’ve run into this a number of times recently. A lot of IOT devices and Pi based devices use pool.ntp.org with no backup. This is always served the same small set of NTP servers advertised by Globe, none of which work.
They should be marked as bad and removed from DNS. ph.pool.ntp.org seems to only serve the same small set of broken Globe servers.
Edit: This issue appears to go back to 2023 and these servers still haven’t been removed?
I have devices from two vendors which both don’t function properly in the Philippines due to this issue. I’m sure there are more
I don’t think likely, but obviously possible. The whole system is too complex to say for sure either way just based on a gut feeling or theoretical mind exercise. The proposal was simple and straightforward enough to easily just give it a try. Pretty much the worst that could have happened is that it does not yield the desired effect. But then, we’d know for sure that some more sophisticated approach is needed. And it is clear that it doesn’t solve the issue all on its own for every zone, e.g., it requires a large fraction of capacity to be available already so that smaller netspeed values result in a sufficiently small share of netspeed, and roughly corresponding lowered load.
It is not that there weren’t enough, and good, proposals how to address the issue in various ways in the past. The challenge with many of them is the amount/complexity of changes they would require, touching and changing core parts of the system logic. With increased effort to implement them, which is the bottleneck. And the potential for destabilizing the system when core parts of the system logic are being changed, in turn needing further effort/time/thought to avoid that, further decreasing the likelihood of implementation.
The proposal to add additional netspeed values below the current minimum on the other hand is extremely simple, just a few additional values in a list that has been modified several times in the past already. Effort would have been limited to a few clicks to accept the PR and deploy it, maybe further tweaking the numbers in the list in the process if there were a strong view, e.g., as to the steps in between. And the proposed change is simple enough to pretty much rule out that it could cause major destabilization of the system. In the worst case, it would not be sufficiently effective in addressing the issue.
Precisely the point: The simplicity and low effort in implementing this, and the low risk, vs. other, more sophisticated and comprehensive solution approaches.
Not sure how that would differ from how people do it today. Certainly I do that whenever I add a new server in a zone I don’t know yet, or where there are external limits such as a bandwidth limit or volume quota where I don’t have an intuition yet as to what netspeed setting would allow me to stay within the limits without wasting capacity.
Same as today.
The unit of the load doesn’t matter, whether one measures it in multiples of bits/second, or in packets per second, or any other unit. The point is that as today, there is at least a rough correlation between the netspeed value, and the resulting load. E.g., halving or doubling the netspeed setting would result in roughly half or double the amount of packets/s or bit/s or CPU cycles, or whatever the limiting metric of a specific system is.
Yes, no change whatsoever on that side in my understanding.
Exactly. And in the worst case, it isn’t, then that server operator is out of luck. But it might still help other server operators with other constraints, and/or in other zones.
And again, the proposed change, and the effort to implement it, and the risk to system stability are so simple/low that it would have warranted at least giving it a try. Especially seeing how long this topic has been open already, and the pain it keeps causing.
I guess the owner and the admins of the project are not personally affected by these issues, so what is the incentive to do something about them?
Just as with the refusal to add servers to an underserved zone that are not from the immediate vicinity. If I were in an underserved zone, I would prefer getting service from a server far away, rather than getting no service, or shitty service. Especially as it keeps getting re-iterated in this forum that distance is not bad per se.
I would happily donate to the project if it meant getting this fixed.
I’m currently speaking with the two affected vendors I’m aware of to resolve this issue (Lockly and Fintic) and I’m working with a local ISP (RISE) to get NTP servers into their next capex spend, which should get the servers onto the local exchange (Getafix)
114 million people in the Philippines are affected by this in some way or another
I don’t think it is a money issue, more one of priorities, and acknowledging that the current way of managing the project is not serving large parts of the community well, neither clients nor (potential) server operators. And by community, I don’t mean this forum, but the overall project contributors and users.
Very good! That would help add diversity to the PH zone of the pool. Not sure it would offset the issues caused by the currently dominant servers not being reachable for large parts of the client population. But it might help adding further servers in the shadow of some bigger ones.
Regarding the current servers, I think it is pretty clear they are blocking large parts of the client population, either intentionally, or inadvertently. The reports in this forum attest to this, but also the score graphs of the servers seem to be rather clear. Not sure there is a specific written rule somewhere, but I think a server knowingly not serving large parts of its local client population to the extent that it is causing noticeable issues would in my view warrant manually removing such a server from the pool (possibly after trying to get in touch with the relevant operator, but that is obviously causing additional effort so shouldn’t hold up any remedial action).
But probably neither the project owner nor any of the admins among them. Or anyone from among the majority of regulars in this forum (myself included).
May be the issue is only with the specific vendor domain not working properly in Phillippines? You may want to check how the NTP service is running from the same location with standard, non-vendor domain, for example 2.pool.ntp.org?
There are currently 10 IPv4 servers registered and active in the PH zone (often, at least one of them has too low a score to be included in the pool, so the count sometimes/often is only 9 or less - e.g., as of right now, 222.127.1.25 has a score of only 3.7, yesterday evening, it was 222.127.1.19 that was out of the pool, etc.). And probing the country zone over time confirms that there are 10 servers - the ones mentioned at the beginning of this thread.
So while the post you reference doesn’t mention any servers explicitly, circumstances strongly suggest that the problem is with those servers, and applicable to the general PH zone, and not an issue with a vendor zone.
Is it sure that those handful servers are supplied to the clients located in the IP ranges of Philippines? I think continent (in that case Asia) servers are provided for low server population country zones.