Agree, the NAT is just a nasty workaround. The NAT creates additional problem; it slows down the IPv6 adaption. The good news is that no IPv6 client would suffer due to rate-limiting. It makes the reflection attack pointless. (Unfortunately, DoS is still there.) Implementing aggressive rate limit is a forward looking thing, it merits only appreciation if someone decides to go that way.
It makes absolutely no sense to not have AAAA records for all pool addresses. If @ask canât see the obvious, heâs beyond persuasion.
1 Like
Please donât be so harsh. The situation isnât that bad as it seems. Vendors are adapting, for example in default chrony.conf file of CentOS Stream the only pool line is the following:
pool 2.centos.pool.ntp.org iburst
So for the CentOS users it isnât matter any more if there is AAAA record or not for 0.pool, 1.pool and 3.pool.
1 Like
If thereâs an occasion to be harsh this is it. The fact that distributions do want IPv6, but ignore the genetic pool address and instead use a specific one indicates that the pool is doing something wrong.
@ask failed when he decided to not return at least one AAAA record for pool.ntp.org to reflect the proportion of IPv6 servers in the pool.
1 Like
Raising the rate limits to accommodate CGNAT opens up pool servers to being overwhelmed, at best, or to DoS attacks, at worst.
How prevalent are DDoS attacks using NTP servers as reflectors? Normal time packets are the same size as time queries, and pretty small, so youâd need to be sending as many packets as you want the receiver to receive, and theyâd cost you the same to send to the NTP server for reflection as it would to send them to the victim yourself. That would seem to make NTP servers rather unattractive for DDoS purposes.
Usually, with DDoS attacks, you want a UDP service that sends back some multiple of the query size, so your victim gets more data volume than you are sending out. There have been some NTP configuration/security bugs in the past that made multiplication possible, but these are hopefully fixed now (and a server operator who doesnât bother with basic security updates likely also doesnât bother configuring a reasonable rate limit).
I donât see how IPv6 would help defend an NTP server from attacks. If thereâs 100,000 nodes sending queries from 100,000 unique IPv6 addresses, or from one IPv4 address, the server has to cope with 100,000 queries either way. If the server chooses not to reply to some of the queries, it has no effect on ingress rates at the server during an attackâonly legitimate NTP clients will move on to another server.
Conversely, an attacker could easily send 100,000 packets over IPv6 and never use the same source address for any of them. If anything, that might incur more costs at the server if the server does rate limiting, because now the server has more client address tracking data to manage.
The solution to the IPv4 shortage is IPv6, not NAT.
I mean, yeah, sure, IPv6 is awesome, we should all use itâŚbut I donât see how âreplace IPv4 with IPv6â could help solve this specific problem, even a little.
1 Like
Youâre only thinking of a DOS attack that saturates the bandwidth, forgetting about other resources, such as routers and servers, which can also be targeted to achieve DOS.
Perhaps you are not aware that IPv6 was designed to be much easier to route, but my point is that itâs trivial to block specific offenders using IPv6 without collateral damage. If it werenât for CGNAT, doing the same with IPv4 would be as trivial.
2 Likes
If all of the pools public DNS records (e.g. 0.eu.pool.ntp.org) provide an AAAA record, the CGNAT issue shouldnât be an issue anymore correct?
CGNAT is a workaround for missing IPv6 connectivity to $something.
Development of the NTP pool has been frozen for years so I am not surprised to see Linux distros starting to move away from it.
Lacking AAAAA records in 2022 is unacceptable.
1 Like
Trivial in what way? IPv6 source addresses are just as easily spoofable as IPv4 ones are. The 100k queries from 100k unique addresses scenario @Zygo mentioned above isnât something you can easily defend against unless the attacker does something stupid.
Nope. Many systems still donât have an IPv6 connection at all, so whether or not an AAAA record is present makes no difference for those ones. Unless they have a public IPv4 address, they will still be going through NAT, either onsite or at their ISP (or both).
But many (really a lot) do.
2 Likes
Certainly. Iâm not saying that full AAAA coverage would be a bad thing; itâs been a long time coming and Iâll be very pleased when it finally happens. As you say, there are an awful lot of systems out there which can use it.
Iâm simply pointing out that full AAAA coverage does not entirely resolve the CGNAT situation, as there are still a ton of systems which lack IPv6 connectivity entirely.
2 Likes
Systems without IPv6 then have IPv4 for which CGNAT is not a common thing.
CGNAT is a very common thing on IPv4, particularly among residential & mobile customers. Providers who did not already hold large IPv4 allocations while also experiencing significant growth didnât really have much choice if they wanted to provide IPv4 service, given the lack of available / affordable IPv4 address space to grow into. The precise numbers are hard to quantify, and there seems to be a lack of easy-to-find recent research to quantify it, but research using data from 2014-2016 indicates that it was already commonly used at that time, and growing rapidly.
What makes you think that is is now uncommon?
1 Like
So what is in scope for this thread are NTP clients.
Iâm from Europe and donât know a single provider that connects IPv4 clients via CGNAT - except as you said mobile devices. Either you get IPv4 or you get IPv6 with a CGNAT bridge into the IPv4 world.
Can you provide some none niche examples? E.g. Large providers who do that or mobile devices that use the NTP pool by default?
Here are some Australian ISPs that use CG-NAT:
https://www.aussiebroadband.com.au/help-centre/internet/tech-support/cg-nat/
Vodafone Germany apparently uses it for all new connections
Spanish FTTH ISP:
And of course, Starlink.
1 Like
- The world is larger than merely Europe.
- The world is larger than your personal experience.
- Many people use broadband delivered via mobile networks as their main connection. Writing off mobile connections as âbut itâs just phones, they donât matterâ isnât reasonable.
- ââŚor you get IPv6 with a CGNAT bridge into the IPv4 worldâ is still CGNAT. Things behind that connection which can make use of IPv6 donât need to worry about it nearly as much, but itâs still there, and still a consideration for the things that canât.
- An example in my country (NZ) is 2degrees; one of the larger ISPs here. They are also a mobile carrier. @trs80 has also posted some examples above.
- Where do you think all the IPv4 growth in APNIC has gone? There hasnât been nearly enough public address space available to allocate one for every connection, but the number of connections has grown massively in recent years. While many of them do have an IPv6 service as well, much of that growth is stuck with CGNAT for their IPv4 access.
- Even in cases where IPv6 is available, that is no guarantee that the NTP clients behind that connection actually support it - particularly for older devices, configuring them to use IPv6 NTP may not even be possible (e.g. Iâve run into a lot of networking equipment with this constraint).
- An example of large-scale mobile use of NTP is Snapchat, which caused the pool some significant grief a few years ago when they released a buggy client.
- See also the research posted earlier in this thread which relates specifically to NTP usage from behind CGNAT: What is a reasonable limit? - #34 by marco.davids
1 Like
So we are talking about clients that would still connect via IPv4 using a CGNAT, if the pool would have full AAAA coverage.
Mobile connections over CGNAT, which are used as main internet connection, which have clients that use the pool are - in my eyes - very exotic configurations. Three if conditions to have such a client connecting and in the end not using the default Microsoft, Apple ⌠time settings.
Superloop and Starlink are for sure good examples, thanks for that.
I personally think that the pool should just implement full AAAA coverage and then new figures can be created. As Marco wrote => This behaviour almost exclusively occurs with IPv4 clients
Nobody knows how many of them will be left once IPv6 is fully supported on the pool side.
Yes.
Why do you think that is very exotic? Any IPv4 pool device tethered behind a mobile connection meets this criteria. Remember that this includes a large number of fixed wireless connections; itâs not just cellphone hotspots.
I agree. But as I said above, this doesnât mean the CGNAT problem is gone. It certainly improves the situation, but there is a long tail of gear and connections that canât or wonât use IPv6 even in the presence of those records.
Nobody knows exactly now many, no. But it is clear that the number will still be significant enough that we need to care about serving them properly. There are far too many common cases where simply having an AAAA record is not the only roadblock to using the pool via IPv6.
2 Likes
In the USA home/business internet over wireless is quite popular. T-Mobile has something like 10 Million home internet connections, all over CGNAT. One of the larger fiber providers here that isnât the big telcoâs is all CGNAT. I mean as someone else said, the world is much larger then your small corner.
1 Like
Well, but what matters for me (and my NTP server) is my corner.
However, itâs clear that I underestimated the usage of relevant IPv4 CGNAT traffic. Looking forward for new statistics after AAAA has been implemented.
1 Like