Far away clients from AWS

1

My server is part of the .de europe zone in the pool.

I did a few checks today and realized that I see a lot of traffic from AWS EC2 instances which are geographically far away. So a lot of US and AP based instances connect to my NTP server and I don’t believe that they have added my server directly.

Just want to check if everything is working fine when it comes to the pools GeoDNS and the Amazon network?

2

What was the way you realized that the clients are far away from Germany? Executing traceroute towards the clients, or geoiplookup of the clients’ IP addresses or something else?

3

AWS contains the region the server is located at as part of the rDNS name. I did a few cross checks with traceroute.

Here are a few very recent examples collected with tcpdump port 123

Expected traffic:

10:01:11.757275 IP ec2-3-66-40-236.eu-central-1.compute.amazonaws.com.22136 > x.ntp: NTPv4, Client, length 48

Unexpected traffic:

10:01:11.845737 IP ec2-52-61-41-166.us-gov-west-1.compute.amazonaws.com.ntp > x.ntp: NTPv4, Client, length 48
10:01:12.005623 IP ec2-3-133-31-231.us-east-2.compute.amazonaws.com.ntp > x.ntp: NTPv4, Client, length 48
10:01:12.218805 IP ec2-13-125-54-88.ap-northeast-2.compute.amazonaws.com.47891 > x.ntp: NTPv4, Client, length 48

As shown here, unexpected traffic from regions far away arrives at least every second.

4

I don’t think you are out of the global pool. So if AWS promotes pool.ntp.org then every once in a while your number will turn up there. Not just on de.pool.ntp.org

I see a significantt number of my clients are from NL as expected. But they also arrive from other parts of the world. Like Japan, ComCast (USA), … and IPv6 addresses I haven’t mapped at all.

5

I’m not sure if I fully understand this and would like to know more. Can you please elaborate a bit on this?

6

GeoDNS checks the location of the client by using GeoIP data and routes the client to nearby servers.

I checked what you are saying and you are right. If I query ntp.pool.org from Europe, I get A records from all over the place. So my assumption that GeoDNS is used the same way was wrong.

7

What are the outbound IP addresses of the DNS resolver you are using? The geoip location of those IP addresses are counting. The geoip database should be up to date also, though.

8

Was there something changed with the pool DNS? I believe so.

I couldn’t reproduce what I saw yesterday => now I only get addresses from servers nearby no matter which resolver I use.

Also tcpdump port 123 | grep amazonaws looks much better now. For the final result in regards to that I have to wait a few days.

9

It is normal to receive NTP requests from Amazon servers located in different countries around the world.
I also get requests from Europe, Asia and Australia, even though my NTP server is located in the US. I mentioned this fact here:

10

This is not that strange.

GeoDNS maps resolver IP addresses to NTP servers in a country/continent zone.

It just takes one client in the US to use a resolver in Europe for this client to appear at you folks NTP server in Europe.

So in short: do not always assume that the clients country of origin are the same as the resolver country of origin.

Now, theare are still extra reasons for this to happen:

  • AWS clients use de.pool.ntp.org directly instead of pool.ntp.org (unlikely)
  • Bugs in the MaxMind database
  • AWS uses resolvers from DE/EUROPE for clients in US (or a bug in the maxmind db)
  • combination of any of them above

We wrote a technical report on GeoDNS, rev. engineering to really understand how it works.
You can download it here

But there are two things that should keep separated:

  • NTP Client’s IP addresses (and their country codes mapped by maxmind)
  • DNS resolver’s IP addresses ( and their country codes)

GeoDNS is a DNS server, it only sees the latter: the resolver’s IP addresses, and not the clients behind it. So say, if AWS for this event use a resolver IP address that was (wrongfully) mapped to Europe by Maxmind, then all clients behind this resolver will be served by EU/or EU-country servers, in regardless of the client country code.

In this way, having US-based NTP clients showing up in your NTP Server is not odd; it just need for these clients to use a EU or DE based resolver.

What I notice doing my measurements is that:

  1. Maxmind’s IP2location DB changes quite a lot from version to version.
  2. The NTPPool folks seem to using always the latest version.
  3. IP addresses not found in the maxmind db are mapped to the global zone.

So maybe that explains why you are first seeing faraway clients and you don’t see them anymore. If Maxmind DB cannot map the resolver’s IP address to a country, you end up in Global zone.

BTW, are you sure you’re not in the global zone? Check your server score at their website:
https://www.ntppool.org/scores/94.198.159.11
if you see a @ at zones, then you’re in the global zone.

GeoDNS mapping

We sort of rev. engineered GeoDNS to understand how it really works. It’s all in the tech report, but for brevity I summarize it here.

  1. GeoDNS get the country code from your resolver IP address, provided by MaxMind IP2location db
  2. It then returns you up to 4 random A/AAAA records from the country subzone.
    • For example, if your IP address is mapped to DE, then you’ll be served with 4 random IPs from the de zone.
    • However, if your country zone is empty, ie, has no NTP servers in it, (and there are 125, when we did the analysis – see table 2 on the tech report. Another example: South Sudan and others), you then fall back to the continent zone.
  3. (if GeoDNS can’t get your country , it sends you to the global zone)

Notes: country and continent zones can be quite dynamic. Either because people add/remove servers , or bc they are kicked out by the pool monitoring system. Check fig5 in the tech report

1 Like