DNS configuration tampering on one of our GeoDNS servers

How? As you need to use key’s that are known between servers and clients.

This is not man in the middle take-over. This is DNS-altering.

As it’s → Client → DNS Give me servers from pool → Pool-altered → Here you have 4 IP’s → Client OK thanks I use them. Now what?

The client doesn’t know what it’s getting from the pool, so there is no way to make DNSSEC more secure then normal DNS.

You forget that the pool-DNS was altered…but running with perfect certificates, just serving wrong IP’s not from the pool-servers but from the attacker.

In this case DNSSEC will deliver the wrong IP’s also, but has the proper certificates.

Sorry Marco, I fail to see how that matters when the DNS is victim itself.

No, it has everything to do with the topic. IPv6 security would help a little to protect the users from such a rogue operator, as temporary addresses cannot be used to reach back to the users after they time out.

The inexcusable refusal by @ask to respond to this need for years exposes the pool users to malicious operators.

Majority of the Internet traffic is still IPv4, how does IPv6 help IPv4?

Regards,

Alan

Someone replied to your post.

| ebahapo
March 30 |

  • | - |

No, it has everything to do with the topic. IPv6 security would help a little to protect the users from such a rogue operator, as temporary addresses cannot be used to reach back to the users after they time out.

The inexcusable failure of @ask to respond to this need for years exposes the pool users to malicious operators.

Every time we talk about this we have to again go through the stats for how much traffic is v6 versus v4 and in what circumstance. I don’t think that’s very useful as no one’s mind is changed by the figures, but maybe we can at least agree that IPv6 use is significant, growing and that also the number of networks that are totally IPv6 with CGNAT for IPv4 is also growing. e.g. the largest mobile network in India (Reliance Jio; 450mil subscribers; 40% of the market).

Implementing IPv6 does help IPv4 because every flow made over IPv6 is one that doesn’t need to go through one or more CGNAT.

It would also make having IPv6-only NTP pool servers more interesting; there would be reduced load on these and the clients they see are more likely to be real individual clients not whole networks behind one CGNAT IPv4.

5 Likes

DNSSEC was designed to protect against manipulating DNS answers ‘in flight’, as @ebahapo called it.

1 Like

If I’m reading the incident report right, the DNS responses were not modified in flight, but the source file from which the DNS server picks the IP addresses for the responses was tampered with. DNSSEC would not have helped much in this case.

Bear in mind that the pool DNS servers aren’t simple slave DNS servers that would serve the same “master” data, a scenario where DNSSEC would make more sense.

1 Like

Take a deep breath, @HAHAHAHA.

Again, I am a supporter of implementing full IPv6 support in the Pool, but not having it is not the root cause of this reported DNS config tampering case.

That’s correct, but the same incident report also states:

When we set up the server, we followed our standard process: full administrative control, firewall rules, locked down access. The volunteer’s SSH key remained on the system from the initial VM provisioning. Later, they asked us to open a firewall exception so they could retrieve personal files from the machine. We made an exception. That was a mistake, and it’s not something we’ll do again.

In other words; hopefully the servers will be hardend and locked down really well from now on.

That leaves us with a remaining scenario; changing the DNS ‘in flight’ as @ebahapo mentioned. And for that scenario, DNSSEC could indeed offer protection.

Support for ADoX is another thing to consider, but adoption for that is still rather low.

PS: I’m aware that NTP Pool authoritative name servers are special and require special treatment for DNSSEC, giving even more reason to absolutely lock down access.

They are tough enough against hacking. No problem there.
@ask trusted somebody that abused his trust.

The damage was low to nothing and it was detected.

He has learned his lesson to never trust anybody again over such vital systems.

He has said so. Can we please stop this discussion? As he won’t make this mistake again, trust me he won’t.

I have been hacked, really hacked, a long time ago, it thought me a good lesson.
The hacker contacted me, then told me how to fix the leak. Also tested my server again and told me it was fixed.
Very nice person, I offered to pay for his help, he didn’t wanted money, but I learned a lot from him.

I’m very sure Ask learned a lot about this. Can we please move on to other more important problems?

As it’s fixed, lesson learned. No need to go on and on.

1 Like

I do agree that this incident should be handled from a security aspect and it’s best to be discussed within the core NTP Pool “committee" but not publically.

I am not a supporter of “injecting” other irrelevant agendas e.g. Ipv6 implementation or DNSSEC into this thread. The server was compromised, it doesn’t matter whether it’s IPv6 or IPv6 or dual stack. The core of this incident is unauthorized access to the DNS server, so DNSSEC won’t help much either.

My two cents:

This incident reminded us just how big the potential impact could be if the NTP pool were compromised, so having a broader discussion about better securing it is definitely worthwhile.

By the way, does anyone know who’s on the core NTP Pool committee ?

1 Like

I think Ask is in it :rofl:

We’re 2-3 people that have been helping Ask maintain the servers/VMs for the last decade(s) but I can’t hardly call that a committee.

Volunteers are often super motivated at first but lose interest after a while, it’s normal human nature but it makes it hard to find people that are: 1) motivated and 2) going to stay around for a while.

2 Likes

I think you identified the root of the problem: our DNS servers are quite hardened but there are some attack vectors that we just can’t stop if the attacker has control of the machine/VM.

That being said, we currently lack the funding to rent all of our own infrastructure. geodns is fairly lightweight but I estimate that it would still represent around $5K-$10K per year in hosting, just for the DNS.

If anyone has established contacts at AWS, Azure or Google Cloud, having a sponsorship from them would certainly allow us to improve our security posture greatly. Right now, one VPS provider is sponsoring us with 4 VPS (we’re really grateful for that) and Ask personally funds several more, but we still rely heavily on volunteer-run servers to maintain the DNS servers.

3 Likes