The pool GeoDNS is not alone in deciding where a request is handled

This is another for the record type posts that may be useful in future reference/discussions. Feel free to discuss here if you would like or just ignore it.

Executive summary: Global anycast DNS providers

Over the years there have been a few topics along the lines of:

  • Why is my server getting requests from outside my zone
  • Why isn’t my server included in zones other than where I put it
  • How does the pool distribute requests to clients
  • Other similar topics

Side note: How did I find this? Do a number of DNS requests from a number of locations in Asia and count the number of unique ip addresses returned = 809. The pool reports Asia currently has 611 servers active.

Some examples of the results are below. The “situation” seems to be happening across all the global DNS anycast providers. It may be they share the cache across locations or at the time of the request the DNS query was routed to a different country to be answered. It is what it is.

Note that the ip addresses grepped for are all in the “@ north-america US” zone but are being handed out to Asia clients asking for Asia ip addresses. I guess that the same happens in other zones/regions. I am NOT saying the is a problem that has to be fixed but I am saying that this should be taken into consideration in future discussions of why the world is not working the way you may expect it to work.

$ ntp-pcr --headers (for below csv line)
1  Timestamp,
2  Tested status,
3  Location string,
4  DNS server used,
5  Hostname asked for,
6  IP address returned,
7  Server Stratum,
8  Reference ID as a string,
9  Leap indicator,
10 Min Error in secs,
11 Root Dispersion in secs,
12 Root Delay in secs,
13 Root Distance in secs,
14 Round trip time in secs,
15 Clock offset in secs,
16 Absolute clock offset in mSec

$ zcat *.gz | grep Tested_OK | grep 2602:81c:1000:2::606 
1789525379, Tested_OK, in-bom-aws, 2606:4700:4700::1111, 2.pool.ntp.org, 2602:81c:1000:2::606, 2, 153.153.36.115, 0, 0.000000000, 0.000625610, 0.007980347, 0.128143119, 0.247054672, 0.006837716, 6

$ zcat *.gz | grep Tested_OK | grep 2600:3c06::2000:a7ff:fe46:9a62
1789521834, Tested_OK, sg-sin-ovhcloud, 8.8.8.8, 2.pool.ntp.org, 2600:3c06::2000:a7ff:fe46:9a62, 5, 168.61.215.74, 0, 0.000000000, 0.010665894, 0.009262085, 0.128716157, 0.226838441, 0.009561919, 9

$ zcat *.gz | grep Tested_OK | grep 66.118.230.14
1789548156, Tested_OK, in-bom-vultr, 1.1.1.1, 2.pool.ntp.org, 66.118.230.14, 2, 216.239.35.4, 0, 0.000000000, 0.032592773, 0.012298584, 0.152220927, 0.226957725, -0.069197194, 69
1789559001, Tested_OK, in-bom-vultr, 1.1.1.1, 2.pool.ntp.org, 66.118.230.14, 2, 216.239.35.0, 0, 0.000000000, 0.025802612, 0.012313843, 0.147833365, 0.231747663, -0.069819516, 69

$ zcat *.gz | grep Tested_OK | grep 23.186.168.126
1789524177, Tested_OK, ph-mnl-lightnode, 9.9.9.9, pool.ntp.org, 23.186.168.126, 2, 132.163.96.4, 0, 0.000000000, 0.000900269, 0.034500122, 0.125499118, 0.214697576, -0.009633393, 9
1789542148, Tested_OK, in-bom-vultr, 1.1.1.1, 2.pool.ntp.org, 159.223.204.192, 3, 23.186.168.126, 0, 0.000000000, 0.001403809, 0.056777954, 0.152215556, 0.244845540, -0.001183121, 1
1789557785, Tested_OK, in-bom-aws, 2606:4700:4700::1111, 2.pool.ntp.org, 2604:a880:4:1d0:0:3:7100:b000, 3, 23.186.168.126, 0, 0.000000000, 0.001617432, 0.056838989, 0.160658474, 0.261243095, -0.008597277, 8
1789561356, Tested_OK, ph-mnl-lightnode, 9.9.9.9, pool.ntp.org, 23.186.168.126, 2, 132.163.96.4, 0, 0.000000000, 0.001358032, 0.034362793, 0.118660371, 0.200241885, 0.004385655, 4
1789563811, Tested_OK, ph-mnl-lightnode, 9.9.9.9, pool.ntp.org, 159.223.204.192, 3, 23.186.168.126, 0, 0.000000000, 0.001464844, 0.057266235, 0.155731520, 0.251267117, 0.008480117, 8
1789578232, Tested_OK, cl-scl-vultr, 9.9.9.9, 2.pool.ntp.org, 23.186.168.126, 2, 132.163.96.4, 0, 0.000000000, 0.000732422, 0.034240723, 0.104589778, 0.173473990, 0.009959997, 9

Nice investigations, this one and the others you shared before.

Some complementary thoughts on this one:

  • A few years back, someone did a study on “DNS watersheds”. I.e., showing which geographic areas are being served from which locations of some of the big anycast providers. Both the slides as well as a recording of the talk are available online. Would be interesting to see whether there is any follow-up investigation, or updates since then.
  • For some anycast providers, the watersheds are not so relevant, because they pass the user’s IP address to the authoritative geoservers, i.e., those should detect rather well the location of the client (mostly limited only by MaxMind data quality). But others, e.g., Quad9 and Cloudflare, in their default services, do not support so-called DNS ECS for transferring the client’s IP address. So here, the pool has no other choice but to use the IP address of the recursor that is querying it as approximation for the client location. Given the watersheds, the recursor might be in another country, so give inaccurate client location. But since many of the big services are run by companies based in the USA, experience shows that many of the addresses they use are geo-located to the USA, even if they are actually being used elsewhere (MaxMind data quality).
  • Caching plays a big role. It needs to happen ECS-aware. That would be interesting to investigate, whether the big services are. I guess Google likely is, but with, e.g., Cloudflare not collecting nor forwarding ECS information, how are they supposed to do caching ECS-aware? And what about other, ISP-run recursors, do they do ECS-aware caching?
  • Ask is running the so-called “mapper”, which was subject here recently. It’s purpose is to precisely collect data about how the geo-location of clients works. Not sure Ask himself has already analyzed that data, but maybe you can get access to it for your investigations.

I wouldn’t have expected this to happen with Quad9.

The GeoDNS server is configured to return the “detected zones” and the resolver IP as seen by GeoDNS (and ECS data if any) if you do a TXT query for _country.pool.ntp.org. It’s possible that some of the GeoDNS servers were started without a GeoIP database available I think (at least that rings a vague bell and we’ve added / updated a lot of servers recently).

$ dig -t txt _country.pool.ntp.org +short
"[2607:f8b0:4023:1004::121]:51365" "104.225.13.0" "us north-america @" "/0" "23.150.41.53" "23.150.41.53" "()"

1 Like

It would be interesting to compare the output of the command when run with the local resolver, vs. querying the authoritative servers directly, such as

dig -t txt _country.pool.ntp.org +short @ntpams1.ntppool.net

Another source of confusion may be when asking for an address of one IP family, but the query itself is done via the other IP family.

E.g., when a node has IPv6, the DNS query is often done over IPv6, even when asking for A (IPv4) records. Or the other way round. E.g., I have a bunch of VPSes that have IPv6, but where the provider only provisions IPv4 resolver addresses.

So when the IPv6 address is misplaced, DNS will return IPv4 NTP server addresses for the wrong zone. Or the other way round.

E.g. compare

dig -t txt -4 _country.pool.ntp.org +short @ntpams1.ntppool.net

and

dig -t txt -6 _country.pool.ntp.org +short @ntpams1.ntppool.net

Here a nice example:

2001:4860:4860::8888: "127.0.0.1:50866" "2a06:a880:4:5000::" "de europe @" "/0" "ham1" "212.12.50.229" "()"
8.8.8.8: "[2a00:1450:4001:c02::126]:49957" "85.121.52.0" "de europe @" "/0" "fra1" "178.215.228.40" "()"
2606:4700:4700::1111: "162.158.93.48:25721" "162.158.93.48" "us north-america @" "/0" "ham1" "212.12.50.229" "()"
1.1.1.1: "104.22.123.197:47526" "104.22.123.197" "de europe @" "/0" "hel1" "62.238.13.236" "()"
2620:fe::9: "[2620:171:fa:f0::10]:54483" "2620:171:fa:f0::10" "gb europe @" "/0" "ham1" "212.12.50.229" "()"
9.9.9.9: "[2620:171:f8:f0:9999::201]:16397" "2620:171:f8:f0:9999::201" "de europe @" "/0" "svo1" "185.209.85.151" "()"
2a03:7900:104:1::2: "[2a06:a880:4:5071::123]:51604" "2a06:a880:4:5071::123" "de europe @" "/0" "ams1" "31.3.105.98" "()"
31.3.105.98: "85.121.52.237:49516" "85.121.52.237" "de europe @" "/0" "ams1" "31.3.105.98" "()"

Each line starts with the IP address of the DNS server queried (IPv6 and IPv4 for Google, Cloudflare, Quad9, and ntpams1.ntppool.net), followed by what the query to _country.pool.ntp.org returns.

Note how the query via one instance of Cloudflare places the server in the US zone, while another query via one instance of Quad9 places it in the GB zone, when the server really is in the DE zone (as all the other responses rightly indicate).

The one-liner to create the above is

for j in 2001:4860:4860::8888 8.8.8.8 2606:4700:4700::1111 1.1.1.1 2620:fe::9 9.9.9.9 2a03:7900:104:1::2 31.3.105.98 ; do echo -n "$j: " ; dig -t txt _country.pool.ntp.org +short @$j ; done

Another example, this time trying to detect the “watersheds” for Cloudflare from a specific location in the MY zone. Many queries rightly place the server in Malaysia, but many also in neighboring Singapore. Thus the following command filters out those locations to see whether there’s any other. And indeed, for a notable number of queries, the server is placed in the USA.

for i in {1..200000} ; do dig -t txt _country.pool.ntp.org +short @1.1.1.1 | grep -ve "my asia" -e "sg asia" ; done

Extract from some sample output:

"162.158.92.26:58376" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:58376" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:52603" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:32990" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:31856" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:31856" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:11536" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:11536" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:57260" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:57260" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:57260" "162.158.92.26" "us north-america @" "/0" "tyo1" "103.170.233.169" "()"
"162.158.92.26:50541" "162.158.92.26" "us north-america @" "/0" "zrh8" "212.25.19.23" "()"
"162.158.92.26:50541" "162.158.92.26" "us north-america @" "/0" "zrh8" "212.25.19.23" "()"
"66.42.49.35:37068" "162.158.92.0" "us north-america @" "/0" "mdap1" "178.128.113.248" "()"
"66.42.49.35:37068" "162.158.92.0" "us north-america @" "/0" "mdap1" "178.128.113.248" "()"
"66.42.49.35:37068" "162.158.92.0" "us north-america @" "/0" "mdap1" "178.128.113.248" "()"
"66.42.49.35:37068" "162.158.92.0" "us north-america @" "/0" "mdap1" "178.128.113.248" "()"
"66.42.49.35:37068" "162.158.92.0" "us north-america @" "/0" "mdap1" "178.128.113.248" "()"
"66.42.49.35:37068" "162.158.92.0" "us north-america @" "/0" "mdap1" "178.128.113.248" "()"

Couldn’t visually see that the server had been placed in yet another zone, but running the test again, now filtering out the US zone as well to see whether there’s any others.

I had looked at these things before, but it is still amazing what new stuff one can discover. E.g., my server in Stockholm is assigned to the EG (Egypt) zone when it queries the pool via Quad9, because MaxMind place Quad9’s backend IP addresses in Egypt, even though other methods such as latency-based geolocation rather clearly place the backend IP address in Sweden.

$ for i in {1..200000} ; do for j in 9.9.9.9 149.112.112.9  ; do echo -n "$i " ; dig -t txt _country.pool.ntp.org +short @$j | grep -ve "se europe" ; done ; done
1 "[2620:171:f6:f0::237]:58477" "2620:171:f6:f0::237" "eg africa @" "/0" "man1" "46.227.203.69" "()"
1 "70.34.198.168:46062" "2620:171:f6::" "eg africa @" "/0" "mdeu1" "161.35.37.118" "()"
2 "[2620:171:f6:f0::237]:58477" "2620:171:f6:f0::237" "eg africa @" "/0" "man1" "46.227.203.69" "()"
2 "[2620:171:f6:f0::237]:58477" "2620:171:f6:f0::237" "eg africa @" "/0" "man1" "46.227.203.69" "()"
3 "[2620:171:f6:f0::237]:58477" "2620:171:f6:f0::237" "eg africa @" "/0" "man1" "46.227.203.69" "()"
3 "70.34.198.168:46062" "2620:171:f6::" "eg africa @" "/0" "mdeu1" "161.35.37.118" "()"
4 "[2620:171:f6:f0::237]:58477" "2620:171:f6:f0::237" "eg africa @" "/0" "man1" "46.227.203.69" "()"
4 "[2620:171:f6:f0::237]:58477" "2620:171:f6:f0::237" "eg africa @" "/0" "man1" "46.227.203.69" "()"
5 ^C
$ geoiplookup6 2620:171:f6:f0::237
GeoIP Country V6 Edition: EG, Egypt