Excessive DNS queries

Server operators
1

After adding a reverse DNS to my NTP server IP, I noticed a significant increase in the amount of queries to DNS type A and AAAA records. After consulting the DNS server logs, I found several requests coming from AWS servers and various ISPs.

I understand that it is common for the NTP daemon to perform reverse queries on the hosts defined as the time source, but in this case it should only increase the number of PTR queries and not A or AAAA.

Could someone explain this phenomenon to me? The subdomain receiving so many DNS requests is only used for the NTP server in the Pool.

2

For what it is worth: we see the same happening and I have the same question. I never understood this behaviour of some NTP-software.

1 Like
3

It depends on the software and config the client runs.
Many NTP-deamons like Chrony try to find the hostname or if they have the hostname they verify it.

I don’t mind one bit, as normally your DNS is cached.
How long it’s cached depends on your TTL setting.

If you have a static IP, you can easily set it to a week or more, then you will see an enormeous drop in requests.
As most DNS today set it to 1 hour, bit silly for static-IP’s.

With a Chrony-client you can turn it off, but I doubt many do.

Change TTL to: 604800 seconds = 7 days = “Very long”

Don’t do this on a dynamic-IP!!!

1 Like
4

How many DNS requests per second do you get?

I don’t think its NTP clients doing reverse lookups. Why would they do that? If they need to refresh the IP address, they have the hostname specified in their configuration. It does not make sense to me to resolve the hostname to IP and back. I’m pretty sure it’s not chronyd.

Some tools that come with NTP clients like chronyc and ntpq do reverse lookups to print the hostnames of currently used servers. There is the -n option to suppress that. But they don’t resolve the names back to IP addresses, so I don’t think that would explain what you see.

I’d suspect some custom monitoring tools, maybe something logging all connections together with both reverse and forward lookups to detect suspicious activity (e.g. intrusion detection).

1 Like
5

Yes they do, if you specify IP-adresses in Chrony, it tries to resolve the domain and shows that instead of the IP.

Look at chronyc sources:

^? ntp-main-1.oma.be 1 6 1 10 -8707us[-8707us] +/- 16ms
^? ntp-main-2.oma.be 1 6 1 10 -3799us[-3799us] +/- 10ms

chronyc -n sources:

^? 193.190.230.37 1 6 3 24 -1508us[-1770us] +/- 8281us
^? 193.190.230.65 1 6 3 23 -1576us[-1845us] +/- 8183us

But in the config I specified this:

server ntp1.oma.be
server 193.190.230.37

You can try it yourself. Yes they do :slight_smile:

You can specify it with chronyd or chronyc works the same, no matter where you specify it.

Another example, other server of mine:

Config: server ntp1.heppen.be

Result in chronyc: heppen.be

That is my correct reverse DNS, so it does do it, else it would show ntp1.heppen.be but it doesn’t.

Sorry mate, they do resolve multiple ways. Don’t know about NTPD, but Chrony does.

As for DNS-requests that I get, sorry, I don’t know, my DNS-servers are at a registrar that provide that service for free and they do not complain. The reverse is at my ISP, they don’t complain either.

6

The reverse lookup only happens in chronyc when you run a command printing sources. chronyd (the NTP client) does not do reverse lookups.

You can try running “tcpdump port 53” and restarting chronyd. You should not see any reverse lookups until you run chronyc sources or similar.

It’s possible there are scripts periodically running chronyc without the -n or -N option, but it is something else what is triggering the second forward resolution described in the original post: pool.ntp.org->$YOUR_IP->$YOUR_HOSTNAME->$YOUR_IP.

1 Like
7

NTP clients do not do reverse/forward DNS lookup pairs. They’ll do a forward DNS lookup (A/AAAA records, hostname → numeric) if they have a host name, but if they are given a bare numeric address they will run with it as long as it keeps replying to NTP queries.

Monitoring software does reverse lookups, and sometimes follows up with a forward lookup, mostly to record spoofed domain names in the logs. The DNS information has to be captured as the packets are received or it won’t be available or accurate some weeks or months later when someone has a reason to look at the logs. There are also a number of IP reputation services available (RBLs aren’t just for email), so you might not be seeing all the DNS queries that are triggered automatically when a packet arrives from your server.

FWIW these A/AAAA lookups really do happen. I get an A or AAAA query matching the PTR of my NTP servers at rates from 0.2 to 0.6 per second for each server IP. Not a whole lot considering the NTP packet rates range from 800 to 3000 per second. The DNS queries come from different IP addresses than the NTP packets 98% of the time, which points to the monitor scenario (i.e. a monitor is passively observing a pool of client IPs, and sends a DNS query for its own information when it sees NTP traffic passing through a gateway).

Edit: it also points to a DNS caching scenario, since most clients don’t issue their own DNS queries across the Internet. :stuck_out_tongue:

1 Like
8

Chrony does both, sorry it does unless you start with -n.
However, it’s normally cached by Bind or whatever.
So it shouldn’t give a lot of requests.

To solve it, goto your DNS-config/provider and change the TTL to a way longer number.
As that determine the caching of the IP/Resolving and visa versa.

TTL is the key to the number of requests, with a static IP you should use 1 week and not 1 hour to cache.

I can not make it more clear then that.

For a busy website 1 hour is often enough, for NTP where 1 query can happen every 12 hours it will not work.

9

chronyd 4.0 does not do PTR queries nor lookup A records based on PTR information. It does one A and one AAAA query for each pool and server directive with a host name, and uses the host name supplied in the config file for those queries. It will never send an A or AAAA query with the content of the PTR record.

chronyc is not part of normal NTP client operation.

chronyd’s -n option disables going to the background. It has no effect on DNS.

Increasing DNS TTL doesn’t help very much as 85% of the querying IP addresses are unique. So a longer TTL would bring 1000 queries per day down to 850, assuming every DNS caches for the full TTL.

10

We receive relatively few PTR queries, but ~75 DNS qps for A or AAAA matching that PTR (we do over 120,000 NTP qps). And also a fair number of queries for the authoritative name servers for it. It’s not a problem for us, hence no need to lower any TTL’s, but it it still interesting to know what exactly causes these queries.

11

Sorry but it does, as a reverse A or AAAA lookup is a PTR record:

Meaning, if you supply an domain as NTP it will do a normal IP lookup via A(AAA) lookup.
But if you supply an IP-address it will use PTR to get the domain.

Could be the client is doing it, but could also be the pool-directive doing it.

A DNS-pool (one way or the other) can contain IP’s as well as other domain-names.
A PTR is done by Bind or some other DNS-program, a program doesn’t call PTR directly, the DNS-server does.

Normal lookup:

nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: google.com
Address: 142.251.36.46
Name: google.com
Address: 2a00:1450:4007:818::200e

PTR lookup:

nslookup 142.251.36.46
46.36.251.142.in-addr.arpa name = ams17s12-in-f14.1e100.net.

Seems Google doesn’t care about PTR as a good PTR should look like this:

nslookup heppen.be
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: heppen.be
Address: 77.109.90.72

nslookup 77.109.90.72
72.90.109.77.in-addr.arpa name = heppen.be.

Look here for more explanation:

PTR’s are not specific called by any program, just depends how you ask a DNS if it will be A(AAA) or PTR.

And yes, Chronyd does give peer information to others and it or they try to resolve it.

As such you will see some PTR’s.
PTR was only invented as some idiots where abusing domain-names for spam and other crap. (and still do).

Question is, why do you run a DNS-server if queries are a problem for you? Let some DNS-provider handle it for you.
My registrar does it for me, free of charge. So not my problem :slight_smile:

12

That is true. DNS names would be nearly useless without automatic lookup.

That is not true. If you supply an IP address, then chronyd will use the IP address and generate no DNS queries at all.

It’s easy to run tcpdump -ni eth0 port 53 to see what DNS queries chronyd is (or is not) issuing. Use port 4460 or port 53 or port 123 and you can see all the traffic it generates, including NTS and NTP. Best to run it in a VM with nothing else and run ‘tcpdump -ni eth0’ inside the VM so you can see all traffic.

Note the -n option to tcpdump–without it, tcpdump will generate its own PTR query for every host chronyd talks to so tcpdump can print the host’s name in the output.

Can you specify which chronyd version you are talking about and what non-default options you have set.

I can’t find evidence of support for a peer information protocol implemented in chronyd on the chrony site, documentation, or in Google, or any evidence of network traffic from a captive chronyd instance that would enable such a feature to work. I did find a reference to a remote management interface which could be used to implement this feature (something like ntpc list-peers), but it was removed in chrony 2.2 (and ntpc access is disabled on almost all public servers so it won’t work with ntpd either).

I can’t find any sign of an A/AAAA lookup using a name obtained from a PTR record from a captive chronyd. This is the behavior being discussed here, and the point Bas keeps missing. Yes, chrony sometimes does A record lookups and at other times uses PTR lookups using names and addresses supplied by the user, but chrony never takes the name from PTR and uses it in an A/AAAA query by itself. This needs to happen to create the effect OP observed–the client can’t send an A/AAAA query for the name of any individual host in the NTP pool because the client doesn’t know any name that would reach the server operator’s DNS server for the A/AAAA records. The client only knows pool.ntp.org or some subdomain thereof, and has to get the name for the individual server some other way (like a PTR query on the server’s IP).

While I was looking for ways chrony could be generating A queries from PTR records, I found NTS. NTS does use host names on certificates, but if you configure a NTS server with an IP address, it’s not possible to validate the name on the certificate because chrony doesn’t have a name, only an IP. chrony could look up the PTR record to find the host name, but then chrony has no way to securely verify the PTR information–it can prove that the client has a valid certificate but not which certificate. In the 4.0 build I tested, when I give chrony an IP address and tell it to use NTS, chrony doesn’t even try–there are no PTR lookups or NTP packets, and the IP address is dead to chronyd even if it is later found as the result of a name (A/AAAA) lookup with NTS.

It’s possible to trigger an A/AAAA DNS lookup on a NTP server-chosen hostname with a NTS-KE reply, but 1) if the OP had set up NTS then presumably they are already aware of it and would not ask this question here, and 2) NTS wouldn’t work for pool.ntp.org servers anyway. Also 3) I’m observing the same behavior and I’m definitely not serving NTS.

If the queries are being generated then somebody has to handle them. It’s barely 0.1% of the total NTP traffic, but presumably even that will be a problem in some cases, e.g. something that generates 0.1% of China’s NTP traffic in DNS queries would crush a smaller DNS server.

The signup page (pool.ntp.org: the internet cluster of ntp servers) mentions PTR records but not A/AAAA records. Those are often handled by different DNS servers and sometimes the A/AAAA server is really tiny. It could conceivably create a problem with connection tracking too–we tell people to exclude NTP from conntrack, but we don’t mention DNS in that context. A server that has 500k pps of NTP could start having conntrack problems with DNS.

If the server is dedicated to NTP, there’s no need to provide a PTR record at all. That would prevent the following A record lookup, and also push the PTR DNS query costs up to whoever’s authoritative for the parent domain in the DNS hierarchy.

1 Like
13

Lets try again…

If you use the pool, this happens to discover the servers:

nslookup pool.ntp.org
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: pool.ntp.org
Address: 162.159.200.1
Name: pool.ntp.org
Address: 162.159.200.123
Name: pool.ntp.org
Address: 193.190.198.10
Name: pool.ntp.org
Address: 78.22.241.24

It does nothing but supply IP-adresses.
Chrony or the client DOES reverse-resolve the IP to display the domain.

I have proven this several posts above.
In order to resolve an IP the PTR is used.

It is NOT chrony or any program that uses this PTR, it’s your DNS-resolver (mostly bind) that does this.
Chrony is simply asking this:

nslookup 78.22.241.24
24.241.22.78.in-addr.arpa name = 78-22-241-24.access.telenet.be.

That my friend is a PTR record being asked. And no it doesn’t matter if you set it or not, as the data-center or your provider does this by default by returning something.
As you can see in my example, the Reverse-DNS isn’t set by PTR else it would return something different.

See here what happens:

and

It’s the same nslookup, but the DNS queries different records.

Chrony and/or it’s clients does the same, and it’s your DNS-resolver that asks the question on either IP or Domain, but the way the information is returned is different.

If I use ‘chronyc sources’ then it will try to display the domain, as such the PTR is asked by Nameservers and not A(AAA) records.

I would not be suppriced if e.g. systemd timedatectl tries to resolve domains for e.g. logging puposes.

14

Look at the OP:

After adding a reverse DNS to my NTP server IP, I noticed a significant increase in the amount of queries to DNS type A and AAAA records

We are not discussing PTR lookups here like the kind Chrony does.

The topic is A(AAA) record lookups that appear after an IP joins the NTP pool. Something (I think we both agree now that whatever it is, it’s not Chrony) is doing a PTR lookup to get the name, then doing an A(AAA) lookup on the name from the PTR.

It’s probably an IDS or gateway-level monitoring of some kind, since no part of a normal NTP client would do the A->PTR->A DNS query sequence (pool.ntp.org->1.2.3.4->ntp.example.com) without NTS-KE explicitly telling it to.

An IDS will see the IP address from NTP packets on the wire, then look up PTR and then verify PTR against A, so the IDS DNS query sequence is PTR->A (1.2.3.4->ntp.example.com, IDS does not do anything with pool.ntp.org).

15

Aruba WiFi documentation suggest using pool.ntp.org. After looking up pool.ntp.org (both A and AAA queries) I see sequences such as this:
0.247328 client 8.8.8.8 DNS 88 Standard query 0x0006 PTR 3.16.82.206.in-addr.arpa
0.251995 8.8.8.8 client DNS 114 Standard query response 0x0006 PTR 3.16.82.206.in-addr.arpa PTR ns1.iu13.org
0.312900 client 8.8.8.8 DNS 76 Standard query 0x0007 A ns1.iu13.org
0.315171 8.8.8.8 client DNS 92 Standard query response 0x0007 A ns1.iu13.org A 206.82.16.3

Possibly related.

16

Sorry mate, you lost me :slight_smile:

17

Aruba Wifi would fall into the “gateway running IDS” category, in this case looking up PTR and A records for its own network traffic. Does it keep doing new A requests over time? (more often than the TTL)

18

The Aruba is sending about 2000 requests per hour, all NTP Pool related Much more often than the TTL.

What IP addresses are sending the most DNS requests?

19

I have one NTP server with a distinctive name in its PTR record, not advertised or used for any other service. These are the top 10 hosts doing an A(AAA) query on that name:

  Count IPv4
    261 142.227.51.1
     49 5.148.43.44
     18 207.195.123.23
     17 142.161.2.50
     15 189.40.216.73
     14 172.253.1.3
     13 216.174.151.53
     13 207.195.123.20
     13 172.253.12.130
     12 163.157.254.25

There’s 2705 DNS A(AAA) queries in this data set covering a period of 9 hours. 1572 hosts sent exactly one A(AAA) query during that time. TTL on the A record is one hour. The top 25 hosts did 10 or more queries, all others did 9 or less. 11.8 million NTP packets in the same interval.

Whois blames a number of Canadian ISPs (fair, the NTP server is in Canada) but also Brazil, the UK, and Google (172.253.0.0/16, appears twice in the top 10). #1 is the Canadian Department of Education.

20

ntpq does do forward lookups starting in 4.2.8.

It will display e.g. "2001:db8::1 (example.com)" if it doesn’t forward-confirm.

(I’ve been meaning to reply since February. I must type very slowly.)

2 Likes