Minor new features on the website

[ask]

I pushed two new features to the website recently.

NTP Checks

NTP checks - run an ad-hoc check of a particular NTP server with results from a few randomly-ish chosen monitors. If an operator is trying to fix a firewall configuration, for example, it might be a quicker way to check if it’s working than trying to add a server (and you can do the check without trying to add the server to the pool…).

Server Points

(working name, suggestions for the feature are welcome!)

The system counts how many times each server IP in the system is returned in response to a DNS request to the authoritative DNS servers and then calculates a percentage (displayed as a permyriad (per ten thousand) globally (all requests from all countries) and for each country (requests from that particular country).

I built this to prepare for the future improvements to how queries are distributed between servers so there’s a way to measure / observe if it’s working better.

Since @HansMayer posted in another thread, I’ll use one of those servers as an example

This says that this server is included in ~5 1/10000th of IPv6/AAAA DNS replies (0.05%) globally and 1031 1/10000th of DNS queries from Austria (about 10%!)

This is all very new, but it’s been fun enough for me to look at that I thought it was worth sharing. Most of the DNS servers have been upgraded to include the logging data needed for this, but there are still some of the ~10 billion daily DNS queries that aren’t being logged with the answer data yet.

Another quirk about this is that it’s counting where users are querying from ignoring if they explicitly are requesting a different zone (say a user in the US doing a query for “ca.pool.ntp.org” will count as “us”).

I’m planning for eventually having the country zones go away as we’ve seen a meaningful amount of misuse or trouble from them and for basically everyone the automatic matching works well (if you disagree I’d love to hear about it).

[NTPman]

@ask

I’m planning for eventually having the country zones go away

Do you plan to remove continent zones as well? (for example: europe.pool.ntp.org)

[ask]

Probably, but I could more easily be convinced otherwise for those.

It’s more viable to have a broader server population on a continent basis to make it work as the user expects (get a server in the region they asked for) while keeping the servers balanced in load.

In the future we will, to balance load, have more countries with servers outside that country. If we keep the country zones this will mean you don’t actually get what you are asking for anyway.

Looking at the logs the overwhelming amount of users asking for servers in a different country / region aren’t doing it because it’s more accurate but rather because their local zone might not work well enough; or out of a misguided attempt at “more redundancy”. The system can do both much better and while managing the impact to the NTP Pool server operators (respecting the “netspeed” setting).

[NTPman]

What about a change that both the country and continent zones just become an alias for the main zone? Let the geodns logic decide what are the appropriate servers to provide.

[ebahapo]

@ask

NTP Checks

NTP checks - run an ad-hoc check of a particular NTP server with results from a few randomly-ish chosen monitors. If an operator is trying to fix a firewall configuration, for example, it might be a quicker way to check if it’s working than trying to add a server (and you can do the check without trying to add the server to the pool…).

This is truly welcome. Though there are similar tools out there, it’s nice to not have to leave the NTP server management site to check a server.

However, please, allow a FQDN to be specified as well, resulting in all the IP addresses returned being checked.

Great job, @ask!

[giovane]

`

I’m planning for eventually having the country zones go away as we’ve seen a meaningful amount of misuse or trouble from them and for basically everyone the automatic matching works well (if you disagree I’d love to hear about it).

So you mean that currently users have the option to use one of the two:

1. the default pool.ntp.org
2. their fav zone of choice, as fr.pool.ntp.org

Your idea is to prevent #2, so users would fallback to the default?

The matching works fine,that’s for sure, But it can be abused (we’ve exchanged some e-mails on that).

I think it is equally important to make sure the populations of entire countries are not served by one or two servers who could go out of sync and then create some big problems. One option is to server from their respective continent zone. The pool has more than 4k NTP servers, but entire countries are currently served by only one or two of these, and I think this country-of-origin/country zone too restrictive method.

( I guess one could assume being served by a NTP server in your country would yield better results, but i did not find this assumption to hold true empirically).

I ran measurements from 131 vantage points from 21 countries that have only two IP addresses int their respective country zone – the servers from the Cloudflare. These include Egypt, Lebanon, Macao, Senegal, and others (that was last year when I measured).

I set those 131 VPs to send NTP request to the most popular servers from each continent zone from the pool, and I took the offset as a metric to measure.

One could think that a client say in Egypt would have bad timing data from a server in North America, due to geographical distance. That is not what we’ve found: we see that most VPs receive good timing data from all continents.

In short: what is the reasoning behind this strict mapping, forcing clients to be served only by NTP serves in their respective country of origin?

[mnordhoff]

Random idea: What if there was a geolocated continent.pool.ntp.org? People in small countries could choose to use a wider group of servers, but it would use geolocation so it would be harder to accidentally abuse servers on the opposite side of the world.

[ask]

@NTPman

What about a change that both the country and continent zones just become an alias for the main zone? Let the geodns logic decide what are the appropriate servers to provide.

Yes, exactly. That’s the plan. All the existing domains will continue to work after the migration period.

@ebahapo

However, please, allow a FQDN to be specified as well, resulting in all the IP addresses returned being checked.

There are a bunch of edge cases involved with that – multiple IPs returned, different IPs in different parts of the world, etc and I didn’t take time to work through what the best choices for all that would be. I’ll give it another go later in the summer to make a new iteration.

[ask]

@giovane

I think it is equally important to make sure the populations of entire countries are not served by one or two servers who could go out of sync and then create some big problems. One option is to server from their respective continent zone.

Yes, indeed. The new measurements of IPs served is preparing for building the GeoDNS configuration so that all countries are serviced by a minimum number of servers (20? 30?) and all the servers are (overall) getting a proportional amount of traffic as well as we can figure out.

@giovane

One could think that a client say in Egypt would have bad timing data from a server in North America, due to geographical distance. That is not what we’ve found: we see that most VPs receive good timing data from all continents.

Indeed. “As close as possible” is more about minimizing the risk of asymmetric routing and dropped packets, not so much for accuracy.

Years ago someone added the first server on some island in the south pacific. It had a round trip latency of more than a second to North America if I remember right, but both the monitoring system and ntpd on a test server found it ridiculously accurate all the same.

@giovane

In short: what is the reasoning behind this strict mapping, forcing clients to be served only by NTP serves in their respective country of origin?

In short: I think we can make the computer do it better than how it works out when humans make the choices.

When people choose specific countries, it’s (best case) either a “no-op” because the system by default would have done the same or it’s an attempt at working around the issues you outlined with underserved countries.

In the worst case it’s causing problems when large user populations choose to use servers in Australia or whatever (for example the snapchat incident some years ago). There are a number of other cases where it’s actively working against our attempts at balancing the traffic, which then can exasperate the problem of the underserved countries.

As you point out all this depends on the system doing a better job with the default zones – that’s an obvious prerequisite! My plan is to have a new DNS name for the new zones and then over time migrate the old names to point to the new one (probably country by country so we can start by migrating things that work poorly now).

[HAHAHAHA]

I apologize if this has been answered somewhere before but why there are 2 permyriad numbers for the same country for my server?

[ask]

@HAHAHAHA The second number is this servers relative fraction of the total “netspeed” for all (active) servers in that country. The number in parentheses is the ratio between the two.

For most countries the ratio is between 0.75x and 0.9x (no matter the netspeed of the server), which I think is an indictor of how many queries are sent to servers in other countries than where the user is. (I think this is from use of explicit regional zones, though I haven’t analyzed this to find out).

The traffic permyriad (the first number) is over the last 48-72 hours (last 3 24 hour periods, including the current day UTC); the netspeed permyriad (second number) is “real time” (minus some ~30 minute caching on the CDN), so the ratio is inaccurate for a few days after changing the netspeed.

[HAHAHAHA]

Thanks Ask for the explanation.

So for my case, does it mean that:

1. Netspeed of my server accounts for 8.58% (2nd number) of total netspeed of all servers in HK zone;
2. While in reality my server has only been sent 4.67% (1st number) of total requests from users in the HK zone?

[ask]

Yeah, that’s right (according to the DNS queries since that’s all I can measure). I think what’s happening is that there are so few servers in HK that the ones with low netspeed get way more queries than expected and those with high netspeed (like yours) get less than their proportion.

The future / planned “v2” zones mentioned above will have less of this problem we’d combine servers from more countries to create a larger server population to choose from (and the “split in 4” numeric zones will be discouraged).

[HAHAHAHA]

Thanks Ask and I look forward to seeing the v2 zones soon!

[gombadi]

Hi

Thanks for adding these numbers. They are interesting.

I had a look in the csv file but could not see these numbers. Is there a way to easily get them so we can graph things locally? It would be interesting to see the figures over time.

I have a server in Singapore that gets 90% of its requests from China (not a bad thing) but was wanting to compare the figures from servers in Mumbai & Seoul which don’t get a lot from another country.

sg 	317.56 ‱ 	10.86 ‱ (29.25x)
br 	5.56 ‱
cn 	4.16 ‱
id 	8.40 ‱
us 	0.33 ‱

[alica]

Some minor issue: the client distribution stats only shows up in www.ntppool.org, not in www.pool.ntp.org.

[davehart]

www.pool.ntp.org is not a useful URL in general. In a perfect world, every server providing NTP service in the pool would also have a webserver on their pool IP address which would redirect requests to [www.]pool.ntp.org to www.ntppool.org, but in practice many don’t. As far as I know, the GeoDNS software serving *.pool.ntp.org doesn’t handle www.pool.ntp.org any differently than “normal” DNS requests for NTP service such as pool.ntp.org, 2.pool.ntp.org, or 2.freebsd.pool.ntp.org.

The result is browsing to www.pool.ntp.org is requesting web service from a random NTP pool service IP address, with varying and unpredictable results.

[NTPman]

The www.pool.ntp.org domain name is handled differently than other names in the pool.ntp.org domain. The IP address asociated to that name isn’t any IP address of the NTP servers in the pool.

[ask]

@alica

Some minor issue: the client distribution stats only shows up in www.ntppool.org, not in www.pool.ntp.org.

Thanks. I changed the Fastly config now to redirect the old www domain to www.ntppool.org (to further encourage that *.pool.ntp.org is only used for the NTP service).