Gradually add/remove server to/from pool in parallel to score increase/decrease

Pool Development
,
41

That is what it previously looked like from my vantage point in Germany, but it isn’t true: Like Quad9, and unlike Google DNS, 1.1.1.1 does not support EDNS ECS for privacy reasons.

From my vantage point in Singapore, I had 1.1.1.1 return pool server addresses belonging to the Singapore and Hong Kong zones in response to queries for the global IPv4 zone. That matches with where 1.1.1.1’s backend server IPv4 addresses of anycast server instances reachable from my vantage point seem located based on a recent GeoLite database version:

11 HK, Hong Kong
22 SG, Singapore

Quad9 additionally returned pool servers belonging to the Japan zone, which matches the GeoLite-based apparent country locations of Quad9 backend server IPv4 addresses:

2 DE, Germany
2 HK, Hong Kong
3 JP, Japan
8 SG, Singapore

I currently believe the two backend server IP addresses being located in Germany are an artefact of my test setup.

I agree as well. At the same time, I also agree that there probably currently isn’t much the pool can do about that.

One thing that has been tried, and found not to bring about the desired effect, was to reduce the DNS TTL in the hope that bursts lasting longer than the TTL would be spread across more servers at least.

Returning more IP addresses would indeed be another potential lever. But I also concur that just as with the TTL values, there are tradeoffs that would likely cause, but also require, a complex discussion.

And as the overall system is so complex, with many parts outside the control of the pool, it would be hard to predict the result of any changes, or when simply trying them out, to properly assess the actual effect they have.

It is looking at another aspect than what the thread was originally intended for. In my view, there are two (often interrelated/interdependent) sides: Experience of clients using the pool, and experience of server operators providing their resources to the pool. (A third view would be that of the pool infrastructure.)

The thread was originally started with a focus on the latter (server operator experience). The topic you brought up focused more on the former (“Some clients might get a reply, but for a number of them rate limiting might kick in.” I.e., clients’ experience.), but is also relevant to server operators (“rate limiting” as an attempt of servers at trying to reduce the load they experience, regardless of whether it actually works or not or may even be counter-productive, or whether it is needed or not).

42

any.time.nl is anycast as I understand it (and I’m sure you know, but for the benefit of others). The gratis GeoIP provider might have queried from near the Ukraine and inferred that location from reverse DNS of ICMP destination unreachable responses from routers in a traceroute.

Of course, anycast means there are multiple actual servers, usually spread around the world, that answer to that IP address. With that in mind I can see how the mistake may have happened.

1 Like
43

A long time ago, over a decade, the stable version of ntpd had a primitive implementation of “pool” which would query the DNS name given and spin up as many associations as there were IP addresses in the response. That might have been 4 IPv4 and 4 IPv6 with the current NTP pool GeoDNS implementation, if they were querying an IPv6-enabled zone like 2.*.pool.ntp.org. This might have influenced the decision to provide only 4 addresses of each IP version.

Given the “pool” code in question hasn’t been in ntpd for over a decade, I hope it’s no longer a consideration as far as the pool operators are concerned.

1 Like
44

An up-to-date GeoLite database indicates the Netherlands as location for the IPv6 address:

pi@RPi-173:~ $ geoiplookup6 2001:678:8::123
GeoIP Country V6 Edition: NL, Netherlands

In any case, it popped up in my tests because, among other zones, it is registered in the pool’s Singapore zone. And the pool geo-located my IP address as in Singapore, likely based on the same GeoLite data (if I understood correctly), and thus provided this address to my client.

The point was that since the pool uses a client’s (assumed) geo-location, derived either from the client’s IP address, or the address of the DNS server querying the authoritatives, to derive the zone it’s from, it will provide only servers from that zone, even when querying the global zone.

45

When the number of addresses per IP version is increased, then (or independently) hopefully, also the limitation of clients to servers from their country zone even when querying the global zone will be lifted.

46

RFC 9523 - A Secure Selection and Filtering Mechanism for the Network Time Protocol with Khronos is a good counter-argument to lifting that restriction. That “Khronos” (previously “Chronos”) proposal imagines a single client sometimes querying hundreds of pool servers simultaneously. I objected strenuously to the proposal on the ntp@ietf.org mailing list before the RFC was published but the NTP working group continued to advance the RFC from draft to publication as an informational RFC.

Pool operators might want to consider joining the IETF NTP working group to see their interests better represented in the future.

47

Still need to read up on that RFC, but “lifting” might have been a bit too strong from my side. “Relaxing” is more like it, so that clients in under-served zones get access to a “wider” range of servers, however those are selected. And servers in under-served zones get relief because they don’t have to deal with pretty much all clients in their country zone on their own, but get support from servers outside their own zone.

1 Like
48

That’s meaningless, though, since we’re talking about anycast IP addresses which are served by backend systems spread amongst a number of data centers, typically around the world. any.time.nl is anycast, as is 1.1.1.1 (Cloudflare public DNS), 4.4.4.4 (Level 3 public DNS, originally GTE if my hazy memory serves, and the granddaddy of widely-used, easily remembered public resolvers, though it was intended for customers originally), 8.8.8.8 (Google public DNS), and 9.9.9.9 (Quad9 security-oriented public DNS).

As an aside, Windows 11 enables easy (manual) configuration of DNS over HTTPS against a number of these, providing DNS privacy from ISPs who may be selling your personal information to data brokers, here in the still-Wild-West of data privacy known to the rest of the world as the United States.

I wouldn’t assume GeoLite. You can check the source code under GitHub - abh/geodns: DNS server with per-client targeted responses

49

All over the planet? Is there any monitor server in Chine Mainland? Because of the firewall between China Mainland and other parts of the world, there is a big difference between observing results from China Mainland internal or external. So if there is no monitor server in China Mainland, it will be a joke to observe from all over the world and get the correct status of a China Mainland NTP server.

50

I am wondering: Is there an actual issue with the monitoring? I think it might have been discussed before, but don’t recall the details/outcome.

As has been stated previously throughout this forum, the pool is a volunteer-run project, both as far as contributing servers are (mostly) concerned, as well as far as the back-end infrastructure is concerned, including the monitors.

So all it takes is a volunteer to provide for the resources to set up a monitor within China. Ask obviously needs to do something to actually integrate it into the overall system, but I’d venture to say that other monitor operators on this forum would be willing to lend a hand in setting up the software itself so as to minimize dependence on Ask, who usually got his hands full with other stuff (obviously not sure about current situation).

I remember that you previously elaborated on why it is challenging for volunteers to set up an NTP server for the pool. I guess much of that could be applicable to a monitoring server as well.

But if the local policy is what it is, not sure how the project can help to mitigate that. Just as with the “firewall between China Mainland and other parts of the world”. As is the purpose of any firewall, if the Great Firewall on purpose limits the free flow of data packets, thereby affecting NTP traffic, both pool monitoring as well as actual time service, I don’t have an idea how the pool could mitigate that.

Or what would be your proposal as to how to address those challenges from the global pool community/infrastructure side? What is your expectation towards the pool in bringing this up?

E.g., maybe as with time service, a big (or even small) company from within China could step in?

Or maybe the volume of monitoring traffic (vs. the volume of time service traffic) is small enough so that hosting such a service on a rented VPS is feasible financially for a volunteer, or group of volunteers?

Or… (your ideas and input needed here - others welcome to chip in as well, of course)?

51

Many good NTP servers was kicked from pool because those servers were observed badly from outside China Mainland.

@felixonmars and @allenZ said that some volunteers have been willing to provide resources to set up monitoring servers in CN for five years, but have not received any substantial response from the NTP Pool project, according to information I got from @felixonmars

Since we return servers by location, the quality of NTP servers should be mainly observed by local monitor servers, or at least use nearby monitors. We need a weighting mechanism to reduce the weight of results from observation servers further along the network path. Note, that all the distance I mentioned above is pointed to “network distance” not “physical distance”.

1 Like
52

Is this still a problem today? And is it sure that this isn’t caused by the servers themselves being overloaded, as can be observed in zones not impacted by the Great Firewall?

Ok, I thought this might have come up before, but wasn’t sure/didn’t follow related discussions in detail in the past.

Sorry to hear that. Unfortunately, this is a bit of a pattern in other areas as well. But if I recall correctly, new monitors have been added “recently”, i.e., certainly since the new monitoring system has gone live, and even in recent months. Also, I remember faintly that questions related to setting up a monitor have been raised in this forum “recently”, and got some feedback. Though the discussion was taken into private channels, so not sure what has come of it. So I can only suggest to keep at it.

Actually, the current system is even better than that. In that it simply picks the best-scoring monitors for a server (with some smoothing so as to not change monitors too often). Thus obviating the need for any more explicit factors to be considered, such as some distance metric.

So you’d need three monitors within the China zone to always have a majority of good servers determine the median score of all five active monitors, and you should always get perfect scores :sunglasses:.

53

It would be nice if the Internet actually worked right, though.

3 Likes
54

Of course. For example, the NTP server provided by our NIM (The National Institute of Metrology, China)
https://www.ntppool.org/scores/111.203.6.13
Sever provide by Aliyun
https://www.ntppool.org/scores/203.107.6.88

Great news! Hope we can have some monitor servers in CN soon.

55

How do you know they are not overloaded (or some “local” infrastructure “in front” of it is)?

Look at this server in the Singapore zone. It is not impacted by the Great Firewall, and my current analysis of the pattern of scores, and how/when it moves into and out of the pool suggests that it is simply overloaded (or something leading to it is, such as a (home) router).

See some people in the thread you referred to having their server sitting outside the China zone added to the China zone, and having to have it removed again, e.g., because the server or the router in front of the server couldn’t handle the load.

56

You do realize this is a project run by volunteers.
What happens in China can not be fixed by us.

I suggest you write a letter to the Communist Party and ask them why they don’t allow China to be part of this project.

Pretty sure loads of people in China would want to join, but it’s China gouvenrment that regulates it.

This got nothing to do with us.

In fact, we would welcome Chinese ntp-servers and monitors.

We can not change the way China does Internet. Sorry.

57

No one ask the NTP Pool community to “kill” the GFW. We just want there is a way to reduce the effect of GFW in NTP Monitor.

First, they doesn’t reject China bacome the part of this project. They treat all cross-border network traffic equally. From DNS to NTP, to HTTP.
Second, please read this article.

You can help we setup monitors in China.

No, you don’t. As I told you in Gradually add/remove server to/from pool in parallel to score increase/decrease - #51 by Coelacanthus
We, especially @felixonmars @allenZ @tuna, organized human and material resources (at that time we had contacted three parties willing to provide servers with GPS) and worked with the community to build a monitoring server in China, but unfortunately, all our common sense communications with @ask fell on deaf ears.

58

We can not fix China, you have to check with the government there.

They are the one that block everything.

Why? I’m not one bit interested in China.

Go there yourself and setup servers that do this, nobody will stop you.

Except the leaders of China…but that is your problem, not mine.

59

No. If there is no ntp pool admin add our server to monitor list, the server has no difference with nothing because it can’t affect the scores of NTP servers.

60

That was not the question. The question was for a specific group of volunteers asking for support in setting up a monitoring server in their area. The specifics are secondary. The area/zone in question is large enough, both geography-wise as well as user base-wise that it would make sense for the pool to have a monitor in that area to assess how clients from that area would perceive servers in that area.

Sorry, but for your purported non-interest, the effective amount of (non-constructive) input seems a bit disproportionate.

A more constructive approach would have been to put your grievances aside and constructively support your fellow NTP pool volunteers in setting up a monitoring server in a zone/area that happens to be China, just as I’d hope you would for volunteers from other places.

1 Like