Excessive traffic to Australian servers?

The system is getting A LOT of DNS queries for “au.pool.ntp.org”. The numbered zones are getting very reasonable numbers each (dozens per second), but the unnumbered zone is getting about 3,000 queries per second.

Do those of you who operate servers in the au zone see a significant number of NTP queries, or is it just some client being really dumb about how it does DNS?

They are all A queries (IPv4), the AAAA (IPv6) query counts are completely normal.

They mostly come from users in the US, Spain, Italy, Brail, Russia, UK, Poland, India, Argentina, Germany, Ukraine, Canada, Japan, Czech Republic, etc. Below is a list of the countries with more than 25 queries a second (in the last 5 minutes, but it seems very weirdly steady) to “au.pool.ntp.org”.

country queries per second
{usercc=“us”} 279.2647329540767
{usercc=“es”} 199.3027877089052
{usercc=“it”} 170.18564873783168
{usercc=“br”} 161.33811896819353
{usercc=“ru”} 127.01876197225846
{usercc=“au”} 106.40284948630855
{usercc=“gb”} 93.28467051126415
{usercc=“in”} 86.46887652652063
{usercc=“pl”} 85.03463476587642
{usercc=“ar”} 81.41922577381254
{usercc=“de”} 68.25097244438817
{usercc=“ua”} 60.15102194654916
{usercc=“ca”} 52.78672309360742
{usercc=“jp”} 50.900799101287795
{usercc=“tw”} 42.38346892206707
{usercc=“cz”} 41.9839509548868
{usercc=“vn”} 37.81754024608716
{usercc=“mx”} 36.40164330396769
{usercc=“hu”} 35.13384399052373
{usercc=“tr”} 30.11717148913948
{usercc=“sk”} 26.300380078442217
{usercc=“fr”} 25.917075927304463
1 Like

Oddly, I’ve actually seen a drop in traffic over the past couple of days to my Australian server.

This is the traffic over the past 7 days, the green line is a 24hr average bandwidth, going from ~2.6mb/sec down to ~1.2mb/sec.
Screenshot%20from%202018-06-01%2013-52-54

Server is in Sidney, with @ Oceania AU pools for both IPv4 (connection speed 25mb/sec) and IPv6 (connection speed 1000mb/sec), and has a solid +20 monitor score.

Unfortunately, I can’t comment on the source of much of the traffic I received, but this is what my ISP’s Internet usage chart looked like yesterday afternoon.

adslstats

I’m expecting a pelican to fly into my email inbox when Internode send me the next invoice.

57

The excess usage totals, for what it’s worth. My usual bill is about AU$60. I’m told they cap excess usage to $300, I shall find out.

After what I’ve seen on the networks I administrate, there are some TP-Link devices that are doing excessive NTP and DNS queries (to the local resolver).
I can’t tell at the moment, if they are asking for au.pool.ntp.org - I could find out in the next few days.
From what I’ve seen I can tell, they are definetily using pool NTP servers. I just can’t tell which pool :wink:

The exact model I’ve spotted is a TL-WA860RE WiFi repeater. I don’t have any details to configuration or firmware version since I have no access to these devices.

EDIT: Maybe related to this one?
Could be a new firmware version which brought back the old behaviour?

Ugh, that seems likely. At Amazon Route53 pricing their ~3000 qps (about 8 billion requests per month) would be $40k/year.

I’ve only recently started using grafana so I don’t have much historical data.

Forgot to mention that the server is also a DNS resolver so the data might be skewed. The server is rented from Vultr in Sydney. I’m using chrony with the default rate limiting enabled.

That figures, someone should show these goons what libpcap does… then they won’t have to burn through peoples’ Internet quotas to see if the Internet is working.

Btw: Is it intended, that the AU zone has a lower TTL (55 seconds) than the other zones (150 seconds)?
Maybe the by 3 times lower TTL is the reason for that many DNS queries?

1 Like

Indeed! I can’t believe I forgot about that. Brazil and Australia both had an override; Australia’s was related to the Snapchat traffic spike. I’ll reset it to normal so we can get better numbers.

Ok, that did decrease the DNS queries for the au zone, but it’s still upside down from the other zones.

Most zones have more queries from users in that country than queries to the “cc.pool.ntp.org” zones. I made a snapshot of AU, BR (which also had a shorter TTL) and CA (just for comparison):

Queries per second for the explicit zone name:

https://grafana.ntppool.org/dashboard/snapshot/8USwAraZFq1feLCkMYiE3pVephU3qTLH?orgId=0

Queries from users (well, DNS servers…) in that country:

https://grafana.ntppool.org/dashboard/snapshot/r5xVgs69F1bi5uPIHYraW7p7Iz7zjCv1?orgId=0

(Not all the DNS servers contribute to these metrics, so the absolute values are of).

Those grafana links don’t seem to work, I get an empty response.

They should be back now. Grafana (and Prometheus) is running in the same Kubernetes cluster as the website and it was unhappy. :frowning:

https://status.ntppool.org/incidents/g59fnjf21dck