Well, perhaps it should :-). If you want the monitor to behave like a client, then it needs to have the same robust algorithms as a real client, … Once again, we seem to be confusing server accessability evaluation with ntp client operation, which are not the same things.
I think your confusion comes from not realizing they are exactly the same thing.
TCP treats packet loss as a signal to decrease the packet transmission rate. If it did not, and every client attempted to retry transmissions at the same or a higher rate rate, a server host would quickly be flooded with packets it couldn’t possibly receive. Classic TCP cut the transmission rate in half every time a dropped packet was observed. TCP must deliver the content of every packet and, so TCP retransmits packets. TCP does not have fixed requirements for when packets are transmitted, so it seeks the highest rate available whenever it has packets to transmit.
NTP uses a similar approach, but with a very much narrower range of transmission rates–just one. NTP daemons need a fixed number of packets transmitted at fixed times driven by the NTP PLL algorithm, so NTP always transmits packets at the minimum (and only) rate required for that algorithm. NTP daemons can lose a few packets and still work, so NTP daemons do not attempt any retransmission–the one packet they send every 17 minutes is all they will ever send (initially packets are sent every 64 seconds, and later sent every 1024 seconds, but this is driven by the requirements for timekeeping and not by network packet loss).
When NTP clients observe packet loss, they will first try to do without the lost packets and use different servers for time sync (the recommended 4 servers includes one potentially dead server). After a very long outage, modern clients will go back to DNS to try to find a different server to bring the client’s working server count back up to 4. While this occurs, the NTP pool should be directing new clients away from servers that are dropping packets, otherwise the client might get the same dead server again.
We want the NTP clients to drop servers that can’t respond. It is the core of NTP’s congestion avoidance algorithm. The NTP pool monitor should use the same packet loss threshold as the clients–if the monitor is different in any way, it will either waste server resources by not utilizing servers that clients would accept, or it will send NTP clients to servers the NTP clients will reject.
NTP daemons only need a 12.5% query reply rate–one reply to 8 queries from the PLL algorithm.
ntpdate requires only 25%. These are extremely achievable targets on the modern Internet. If an NTP server can’t deliver that, something is wrong with it, and it should be removed from the pool until it’s fixed. Common examples of such problems include the application of stateful firewalls that cannot track all the NTP client sessions, or the use of a hosting provider that aggressively discards NTP packets, or a bandwidth setting several orders of magnitude too high.
Unfortunately, the current NTP monitor itself has problems hitting this target with significant parts of the Internet. This needs to be fixed for the NTP pool to work properly.
Statistically, that would have much better odds (yes, that is the right word) of making contact over a noisy channel.
The point of monitoring is to remove servers from the pool whose channels are too noisy to be reachable by standard NTP clients. The point of the single-query-single-response NTP design is to keep the channels free of noise from retries and other non-time-synchronizing packets.
The current monitor has only one channel, and it’s full of so much nearly continuous noise. We can try to measure that noise and try to subtract it from the signal, but that’s some pretty heavy signal processing, and the active sensing required to make it happen from a single network entry point will itself add more noise in the same channel.
If we used multiple channels, we can easily and reliably correlate them to extract signal, without adding enough noise to change the signal. The trick is to build a diverse channel set, but that’s easy to do with volunteers who live in diverse places setting up monitor nodes near themselves.
As for servers blocking fast polls, there are only retries until a reply, with the rest either assumed to be dropped or lost in either direction, so pretty much irrelevant.
This is not how blocking works. If the first reply is lost, and the client retries, the server will not send another reply until the client’s sending rate drops to an acceptable level. Each retry restarts the ban timer.
Since it’s not possible to duplicate the variable path quality as each of thousands of clients, you have a to eliminate that common variable in the equation to get a meaningful result from monitoring.
It’s not necessary to duplicate every possible path, just a representative sample of them that is big enough to keep the error rate below an acceptable threshold.
Like trying to decode a signal in a noisy channel, you only see part of the picture otherwise
This is like trying to take a picture of our house from across the street, but there is a tree in our yard blocking our view. We can take a lot of photos of the tree in front of the house, hoping the wind will blow leaves and branches around so that we can see some part of the entire house in each photo, and photoshop them all together to get a picture of the house, but to do this we need to be able to figure out which pixels in each individual photo are tree and which are house, which is an AI research project. The trunk is probably not going to move very much with just wind, so there are parts of our house that we can never see this way, even if we solved all the other problems.
We could do all that, or…we could just pick up our camera and move it so the tree isn’t in the way of our photo.
Most client network paths are very similar. Most of the reachability problems on global scale occur in a handful of places (like underwater cables and oversubscribed/underfunded network peering links) so we only need to have two nodes on either side of those. We don’t even need to know where those problems are in the network–we can discover them automatically, and filter them out of the monitoring signal, by analyzing the monitoring data.
A couple of years ago, during another conversation about this topic, I built a proof-of-concept multi-monitoring system. I recruited a number of monitoring hosts with connections as diverse as I could beg, buy, borrow, or steal. I included residential broadband access points, multiple national carriers, carriers in multiple nations, nodes in business offices with commercial regional ISPs, nodes in friends’ and relatives’ homes, nodes in data centers. I purchased a few days of VPS hosting in data centers around the world. To simulate a “stolen” node, I set a monitor up in a relative’s apartment, using a coffee shop’s open wifi across the street (I did obtain permission to use this network beforehand, but the technical implementation would have been identical if I had not).
I had my pool of monitors monitoring a set of 50 NTP servers around the world, randomly chosen from the NTP pool so that I would have NTP pool monitoring data for comparison. I put their server scores from the NTP pool monitor and my own NTP monitors in a grid with a NTP monitor in each column, a NTP server in each row, colored each cell according to how reachable that monitor thought that server was, and made a stack of such grids over time.
Correlations along rows indicate server-side outages (every monitor sees the host is down because the host is down). Correlations along columns indicate failing or redundant monitors (the monitor thinks every server is down because the monitor is disconnected from everything). Inverse correlations along part of a column indicate a net split–monitors on one side of the split disagree with monitors on the other side of the split about the reachability of servers on the other side of the split. Noise indicates network congestion (can appear in either columns or rows, which identify which end of the path is most affected). Statistical algorithms can automatically analyze the data and recommend places to prune or expand the monitor pool, but given the small size of the data set I just turned the data into an animated visualization and analyzed it with my eyeballs.
I went through a few cycles of pruning redundant monitors whose reachability data correlated too strongly with other monitors, and spinning up monitors in new locations to bring the total back to 10. I kept monitor nodes that could reach servers that other monitors could not reach, and discarded monitors that could only see the same servers or fewer servers than other monitors. After two weeks I had 13 monitors returning 3-5 unique reachable server sets (depending on where the threshold of uniqueness was set and the time interval compared).
So we definitely don’t need thousands of monitors for the NTP pool–half a dozen could be enough, if they are in the right places. Double that number keep things working when there are server failures and changes in global network structure (over time, some monitors will become redundant as network paths merge, some new paths will be created requiring monitors in new locations to test them). If monitors provide duplicate results, discard all but the cheapest one and try new monitor locations.
I ranked the NTP monitors by how different their scores were from the consensus reachability (the highest reachability score for each server observed by any 2 monitors). The NTP pool’s production monitor was ranked 13th out of 13, i.e. it was the worst.
If we’re looking for a quick and easy fix for the NTP monitor, one possible solution is to move the NTP pool monitor to any randomly chosen data center in North America. 93% of the monitor locations in my experiment were better than where the monitor is now, and I was trying to find bad ones. I put “stolen wifi” in my monitoring pool because I was trying to build a monitor that was worse than Newark…and failed (I could have achieved it with an
iptables rule to create 50%+ packet loss, but that would have been cheating).
ntpdate was deprecated, so how much support should ntp provide for obsolete facilities ?.
I mostly agree with you here, but there’s still a lot of single-shot NTP traffic in the pool (maybe not
ntpdate any more, but smartphone apps with ntp query libraries embedded in them are apparently common).
It doesn’t matter much whether the cutoff is 75% unreachable (NTP daemon) or 87.5% unreachable (
ntpdate)–both are extremely high packet loss rates by current Internet standards.
An ntp client effectively doesn’t care if one of it’s basket of servers goes down for a while, so why does it need to be so draconian
Clients do care. Dropping a server with a true packet loss rate over 50% isn’t draconian. If it’s much higher than that, clients will drop it themselves.
Also note that if monitoring is done correctly (along diverse paths) it’s not very draconian at all, because single-packet queries do accurately reflect server reachability if you are doing them from somewhere near the intended clients (like on the same side of the planet). Also, each monitor sends its own query packet, so we would get those extra tries to get a response that you were asking for, just in a much more useful way.
When it’s done incorrectly (from a single location next to a bad peer), a lot of good hosts get kicked out of the pool. The current NTP pool monitor was able to reach about 5% fewer servers than the other monitor nodes in my experiment. Traceroute said packets to all of the failing NTP servers were sent through the same peer right next to the Newark data center.
Of course there are lots of other problems with my experiment–too short, not enough nodes, not enough continents, I ran it two years ago instead of this week…but it’s enough to see the shape of what we are not seeing when we monitor the pool from only one point.
I wonder if there is a case for a self contained, standalone keep it simple monitor application
For my experiment I just put all the NTP servers in chrony, scraped reachability data from
chrony sources, and used some perl to turn it into tables. Not the most efficient way to do it, but I only had a million cells of data to worry about, and I didn’t want to spend more than a few hours to build the data collection and visualization tools.
NTP server on an island…
No system should be expected to deal with extreme cases like that
Well, I agree it’s unsolvably hard for the NTP pool to do it with DNS service; however, it’s a trivial problem to solve using an anycast service. If, everywhere on the Internet, 123.456.78.90 was the IP address of a nearby working NTP server with +/- 50 ms accuracy, then the island just needs to drop one NTP server there, and configure its routers to rfc1546 it, and nobody needs to talk to the NTP pool’s DNS server. OK, we’d need 4 IP addresses, but you get the idea.