DNS configuration tampering on one of our GeoDNS servers
We found that a volunteer who provided hosting for one of our GeoDNS servers used their access to manipulate DNS zone weights for the NTP Pool service domain. The server has been secured and removed from the DNS NS-set.
What happened
One of our geodns servers (ntpmnl1, in Manila) was hosted on a VM provided by a volunteer. When we set up the server, we followed our standard process: full administrative control, firewall rules, locked down access. The volunteer’s SSH key remained on the system from the initial VM provisioning. Later, they asked us to open a firewall exception so they could retrieve personal files from the machine. We made an exception. That was a mistake, and it’s not something we’ll do again.
The volunteer used that access to install a tool that modified the geodns zone data every two minutes, boosting the AAAA record weights of 42 specific IPv6 addresses, all NTP servers the same person had registered in the pool. Many of these servers were already active in the pool in Asian countries, the US, and Mexico.
They also installed a reverse proxy tunnel for persistent remote access and ran a packet capture tool to log IPv6 source addresses of DNS queries hitting the server.
Our configuration system refreshes zone data on geodns servers regularly, so the modifications were overwritten each time. The tool ran every two minutes to re-apply them between refreshes.
What was the actual impact?
The impact was limited. ntpmnl1 was one of many geodns servers and handled only 2-10% of AAAA traffic for any given country. The zone refresh cycle also kept overwriting the modifications, though the two-minute cron was often fast enough to re-inject before the next refresh. Some countries saw clean stretches of several hours (DE had a 5-hour gap on March 19), but it wasn’t a regular pattern.
For the first ~41 hours the tool only boosted weights of IPs that were already present in zone entries, similar to what an operator could do by adjusting netspeed on the pool management website. For countries where these servers weren’t registered (France, Poland, Sweden, Argentina, Nigeria, etc.) the net effect was effectively 0% of total AAAA queries, at most 0.5%. For countries where the servers were already registered (US, SG, AU, BR, etc.) the net effect was higher, around 0.5-7% of total queries, since ntpmnl1 handled 2-10% of each country’s traffic and the volunteer’s servers were heavily favored in its responses.
Later versions of the tool tried to inject IPs into zone entries for countries where the servers weren’t registered. One test ran for 68 minutes before being reverted; the most aggressive version ran for about 20 minutes before being discovered. Neither had much time to take effect.
Even in affected responses, clients usually got a mix of the volunteer’s servers and regular pool servers. In the US, about half of affected queries had all 4 answer IPs from the volunteer; the other half included at least one normal server. In Canada, most affected responses had 3 regular servers and 1 from the volunteer. NTP clients using multiple servers (ntpd, chrony, systemd-timesyncd) would still have had unaffected time sources available.
All 42 servers are registered pool members. Most have the maximum monitoring score of 20, and we have no data indicating NTP queries to these servers weren’t answered accurately. The servers that were most prominent in affected responses matched their labeled geography: US-labeled IPs dominated US queries, SG-labeled IPs dominated Singapore queries, and so on.
None of that changes the fact that what this volunteer did was wildly inappropriate. Tampering with DNS infrastructure that hundreds of millions to billions of devices depend on, regardless of whether the NTP responses were accurate, is a serious breach of trust.
How we run our DNS infrastructure
Our policy is to maintain complete administrative control of our DNS servers. We don’t give outside access to anyone. The servers run either on infrastructure we acquire commercially or on machines hosted by long-standing, trusted community members.
The NTP Pool runs on limited resources. We depend on the community to help us operate, and that means trusting the people we work with. This volunteer had been a pool contributor for over a year, running NTP servers in parts of the world that are poorly served. That track record is why we worked with them and made the exception on the firewall.
Going forward, we’ll be more careful about who we work with on hosting, and we won’t be making exceptions to our access policies. If you’re a long-time pool participant and want to help with DNS server hosting, have a look at pool.ntp.org: NTP Pool DNS servers .
What we’re doing about it
- The server has been secured and removed from the DNS NS-set
- We’re reviewing access controls across all geodns nodes
- We’re looking at the pool account that registered these servers
- We’re adding integrity checks for zone data on geodns servers
- We won’t be granting firewall exceptions for host access going forward
Timeline (UTC, March 2026)
- March 14: Volunteer installs a reverse proxy tunnel on the server for persistent remote access
- March 17-18: DNS packet capture begins
- March 18 08:16: First zone file modification. For the first ~41 hours, the tool only boosted weights of servers already present in zone entries, similar to changing netspeed on the manage website. Impact on countries where the servers weren’t registered was under 0.1%.
- March 19 10:54: A 68-minute test of a more aggressive version that injected IPs into previously empty zone entries. Reverted afterward.
- March 20 07:42: Most aggressive version starts, but only runs for about 20 minutes before discovery.
- March 20 09:00: Data clean across all countries. Server secured and removed from the DNS NS-set.
