The time has come: we must enable IPv6 entirely

Hi mlichvar,

Thank you for your input. What I like about it, is that you leave out the difference between IPv4 and IPv6 and rightfully turn this into a more general issue of (suddenly) unreachable servers. You are quite right that this is not exclusively an issue with IPv6.

I’m not sure if introducing an entirely new zone will accomplish that on the short term. It’ll take some time to be adopted. Maybe a lot of time?

Clients that have this ability do not need another zone, I think. They just need to change their ‘server’-directives into ‘pool’-directives (or whatever the syntax is in your software). When this would happen, everything you wish for should already work.

So, as far as I’m concerned, the ‘pool’-directive should be the recommended way in the NTP pool documentation. The only reason it apparently isn’t, is because very old NTP-software doesn’t support it (see also the pool section in this discover.html). The NTP pool documentation could probably benefit from an update that better reflects this. Also, these older versions are becoming increasingly rare nowadays.

Anyway.

Your are obviously quite right that ‘sticky clients’ are a problem, but the way I see it, adding another zone won’t fix that. If we would ever want to introduce a new zone, maybe it is better to introduce ‘[0123].ipv4.pool.ntp.org’ (or, even better perhaps; ‘legacy.pool.ntp.org’)? Serving only A-records as a courtesy to support broken clients, while the existing zones gradually evolve into serving A and AAAA everywhere. :wink:

I doubt that adding IPv6 to all zones would break NTP clients. The old clients which are looking up only “A” type DNS records would not see that change. The newer clients which are looking up “AAAA” type DNS records are already supposed to be prepared to access NTP servers over IPv6.

I do not exclude that there might be some clients in the second group, which are broken, however that must be a bug in the client, and it supposed to be fixed there. That should not stop, not even slow down the introduction of AAAA record type into all of the pool names.

After thinking about it a bit more, I changed my opinion we should implement that change on all names in one shoot and forget about it.

If we introduce the change in smaller steps, what kind of feedback do we expect to correct our direction?

2 Likes

Yes, it would take a long time, but I’d prefer that over breaking existing setups or waiting for all clients to be ready.

If you require some change to happen on a large number of systems, I think we can be pretty sure it will never happen completely.

Yes, it should, but it is not. The currently recommended client in the recommended configuration doesn’t work well with IPv6. I hope you can see the problem we have here.

It absolutely would break some clients. ntpd configured with the server directive is a good example. It doesn’t switch to IPv4 if IPv6 doesn’t work (e.g. due to a network misconfiguration or firewall). I’m sure there are many others.

Actually this is a very interesting thought.

Surely adding AAAA’s to ‘pool.ntp.org’ seems a perfectly safe thing to do, because that name is often used in conjunction with the ‘pool’ directive or with clients like ‘ntpdate’, and all of those a perfectly able to recover from unresponsive IP addresses (both v4 and v6).

It’s not all that bad.

For instance, Ubuntu already ships with the pool-directive nowadays. With a fallback to ntp.ubuntu.com (which also has IPv6). Debian also ships with ‘pool’.

By default, Chrony on CentOS uses the 2.centos.pool.ntp.org as the time server (as you are probably well aware of :wink:). And for Android it’s the same thing. AVM (manufacturer of Fritz!Box CPE’s) ships default with 2.europe.pool.ntp.org and indeed in our Frankfurt (D) node we see an amazing 8-10% of all NTP-traffic coming in via IPv6, primarily because of that default (Fritz!Box is a populair CPE in that region).

Situations where IPv6 doesn’t work, are becoming increasingly rare. Since many major sites have turned it on, a problem doesn’t go unnoticed that easy anymore.

Many cheap embedded devices don’t even support IPv6 and won’t ask for AAAA in the first place.

Most sntp-clients are able to switch to IPv4 if IPv6 doesn’t respond.

And so on.

Also, nowhere do we state that the pool will continue to support broken setups forever, even if it hinders progress (on the contrary; the site warns not to use the pool if lives depend on it).

I dare to say that adding an AAAA to 1.pool.ntp.org will not make the problem bigger than it already is, but it will attract some more IPv6-traffic, which is at least a step forward in the right direction. And doing this 10 years after adding an AAAA to 2.pool.ntp.org would make this a nice milestone also.

I doubt that. If broken IPv6 was so rare, the “Happy eyeballs” algorithm wouldn’t have to specified and implemented in various applications.

Switching 1.pool.ntp.org will cause ntpd clients following the recommended configuration in a broken IPv6 network to have only two working servers. That’s worse than having three or only one. If it needs to be switched, it’d be better to switch two of the numbered zones at once.

1 Like

One more observation: this discussion focusses on IPv4-centric nodes that might have an IPv6-issue, but we are gradually moving towards a situation where the opposite also applies.

Here’s an example. It’s about IPv6-only servers and psSense:

Ticket 9931: 0.pfsense.pool.ntp.org doesn’t work on IPv6 only installations

I find that interesting to be aware of. IPv6-only nodes are becoming real.

Also, there is this news:

4 Likes

The problem is there, however that is not IPv6 specific thing. I just made a test with ntp-4.2.6p5: for a given server line the name resolves to multiple IPv4 addresses (and no IPv6 address). Some IP belongs to working NTP servers, some are to non-working. The ntpd randomly selects working or non-working NTP server. It does not care about reachability for the selection of IP address.

I made another test, it is a bit unrelated to the problem you mentioned, the outcome is very reassuring. The name used on the server line had an IPv4 address and an IPv6 address associated. The test host had working IPv6 stack (nicely reachable via link local address), however there is no global scope route advertised (no Internet reachable over IPv6). The ntpd always selected the IPv4 address even if on that IP address there was no answering NTP server.

After these tests I feel very safe to add AAAA records for all the names of pool.ntp.org.

2 Likes

It was my understanding that an hostname entered will lookup the IP and the hostname isn’t used anymore.
Some forummember mentioned this to me.
As such, there doesn’t need to be any AAAA-records in the pool, just IP’s.

Could it be the forummember is wrong and there is a DNS-lookup all the time?

Because if you look under manage, it will provide the IP after being added.
Is this a wrong assumption?

The pool management is fully IP based. However, if you add a server via name, the pool management software will do DNS lookup once, and stores the servers via their IP address as key. The original name you used becomes just a tag on the servers.

The ntp clients with the server configuration line looks up the IP address from the name once at startup. Then, it will stick to that IP address till the NTP daemon runs. Contrary, with pool configuration line it is much more dynamic. Servers my come and go as their reachability changes. I guess, that involves regular name->IP lookups.

1 Like

That shouldn’t normally happen with the pool. The pool is monitoring servers and should give only addresses that are responding. If ntpd is configured with 4 servers (and 3 resolve to IPv4 addresses), there is a very high chance at least one will be responding when it is started. It needs to be restarted later when the servers are dropped from the pool.

That’s expected, but it is the system resolver doing this. It sorts the addresses per RFC 3484 and ntpd just uses the first returned address. The resolver can check only the local routing configuration. It does not know if your router is misconfigured and announcing a global IPv6 route when it has no upstream IPv6 connectivity, or there is a firewall blocking NTP on IPv6.

An NTP client that tries all addresses sequentially (e.g. older sntp versions from the ntp distribution) will be slowed down significantly if IPv6 doesn’t work. It’s not completely broken, but I’m sure some users will be unhappy if their appliances suddenly take much longer to boot.

If I do an nslookup, I see no changes just IP’s.

Based on IP’s it will be hard to make a pool that is IPv6 only.
Then it better changes to domain only, so you can list both IPv4 and IPv6, but DNS isn’t organised that way.

But beware, Marco is only IPv6 minded, his own website is IPv6 and can’t be accessed by IPv4 at all.

Name: forfun.net
Address: 2001:980:5270:1:83:163:210:97

He’s pushing IPv6 against all odds and ignores the problems people have to implement it.

He’s not here to better NTP, he’s just here to push IPv6, like he does on many websites.

Google for his name and add IPv6, then you know what he’s doing.
He does not care it works fine the way it does, he want’s IPv6 being pushed forward, nothing else.

My test was rather about the NTP client behaviour of NTP server selection depending on reachability of the server, and not about the way the pool is functioning.
However, in reflection of your observation, the pool is monitoring servers the same way for IPv4 and IPv6. Unreachable IPv6 based NTP server will be removed from the pool the same way as the IPv4 ones.

Router misconfiguration may unfortunately happen. That has not much to do with NTP specifically, it most likely affects other kind of traffic too. If the filtering of NTP traffic is incorrectly in place, the router administrator supposed to fix this.

1 Like

On the other hand; things might be broken already right now. If you look at the DNS-statistics on https://status.ntppool.org/, you will see little peaks[1] on the top the hour and some smaller ones every half hour. My guess is that these are cronjobs run with some sort of sntp lookup. Now think of a large ISP doing Carrier Grade NAT. All of a sudden these cronjobs are triggered, the ISP’s resolver resolves something like pool.ntp.org or 0.pool.ntp.org and returns (only) 4 A-records. The same four addresses will be used by all these clients (at one given ISP). Many clients will, at the same time, try to connect to them all at once from only a few IPv4 source addresses (because of CGNAT). There is a fair change this will trigger rate limiting. With AAAA-records in the DNS-response this could have been prevented, because there would be much more unique source addresses.

What I’m trying to say is: do not assume everything is working smoothly right now. The current setup is developing shortcomings too in an ever changing world and also causes some ‘brokeness’ here and there.

[1] The small peak probably represents a multitude of queries due to DNS-caching at the resolver level.

The part you may be missing is this:

Once the server is added to the pool via ‘manage’, let’s say with both an IPv4 and an IPv6 address, the only way a pool-client can thereafter reach that server on it’s IPv6 address is when there is a corresponding AAAA-record stored in the pool-DNS. If only A-records are stored, clients will solely reach it via IPv4.

Only 2.*.pool.ntp.org returns AAAA-records on DNS-queries at the moment. My proposal is to extend this to some of the other names as well (actually it’s not my proposal, I’m just recalling what has been proposed quite a few times before).

This benefits clients who wish to do IPv6 (and servers who wish to receive a little more traffic via IPv6).

A client that wishes to stay at IPv4, simply does an A-query. Nothing will change. This is the default behavior when IPv6 is not enabled on a client.

A server that doesn’t want to receive any IPv6 traffic, simply doesn’t add it’s IPv6-address to the ‘manage’-dashboard (if it has one in the first place).

Nobody is forced to use IPv6.

… and it will be another week where nothing will change. Nobody knows @ask here ? What is the plan ? Seriously he has to say something, No progress on IPV6 since 2013 … The community is supposed to help so do a move on IPV6 or tell us why nothing is done. What is the real problem ?

1 Like

FWIW I agree. Where is ASK and what does he think?

2 Likes

In my opinion, even potentially breaking changes can be done if a proper notice about upcoming changes is published in advance. If I was in charge, I would publish an announcement no later than in October with the following information:

This would give people sufficient time to adjust their configuration if needed.

1 Like

I favor increased IPv6 usage everywhere.
I wonder how/where one publishes notifications that will reach even a subset of the hundreds of millions of NTP Pool users.

ntppool.org has a “News” section. Once an announcement is published, a link to that page can be shared to news sites, Slashdot, Reddit, what have you nowadays. There’s also a mailing list (although it may have issues at the moment but the subscriber list is probably still intact if the server comes back online some day) and the comp.protocols.time.ntp newsgroup. It may be true that we may never reach all the end users, but I’d think we’d be able to reach a significant part of device manufacturers and ntp software authors, who could then figure out if the change affects their devices/software and inform their own end users accordingly.

I don’t think disseminating the news is the biggest problem here. There would need to be a decision about the planned dates first. What I posted above was only my suggestion.