BACKGROUND
My GPS+PPS disciplined NTP server is being monitored by the pool.
I read ntppool uses SNTP for monitoring, which does not take into account asymmetric latencies, but I was curious to know why some candidate monitors are calculating negative scores when all the other monitors are calculating at or near 20?
I don’t think compensating for path asymmetries is a feature of NTP itself (vs. SNTP), but rather a feature of specific implementations. Though indeed keeping some history, like “full” NTP implementations do, would arguably be helpful in doing so, with SNTP clients thus less suitable from that perspective.
I guess that is just a “feature” of the wild, big Internet. Based on own experience, there’s always issues somewhere. In this case, the packets simply might take a route that is overloaded, so a sufficient number of them get dropped for the score to be affected. In some cases, there simply isn’t any connectivity between customers of certain providers because they can’t agree on connecting to each other. E.g., Cogent refusing to peer with Hurricane Electric/HE not willing to pay Cogent for peering prevents any and all IPv6 communication between the two. IPv4 traffic between the two finds a path, but it is far from optimal, potentially impacting NTP traffic.
Normally you only get monitors assigned that score you good.
Those will become active for your server.
All others will be ignored.
Your score only drops when active monitors monitor you badly, typical when your server is online this won’t happen…except when your are working on something that it can’t be reached or your server is ticking wrongly.
Other then that, your score will be perfect and monitors that see your server are selected.
When you add a new server to the pool, it also takes some time for monitors are assigned to your system.
From what most of us see, this system works quite well.
Keep in mind, only the ACTIVE monitors determine your score, all others have NO impact, regardless their score.
And yes, active monitors can be added when candidates score you better.
And yes, active monitors can be removed when actives score worse then candidates.
Normally only the best monitors are assigned to your server. It’s an algorithm that determines this automatically.
The task of a monitor is to see if you are online and ticking correctly, the ms you see is a ping-time not important. A monitor with a higher ping can be very good for your server, while a close monitor may not even reach you.
Also know, the monitors themselves are also checked by the system and internally by the monitor-deamon itself, before it started checking other servers, the monitor checks the system-time of the machine, when it’s off, it will stop checking others.
Not sure what the issue with “false” negative scores is, apart from an “optical” thing (doesn’t look good in the graphs). I wouldn’t even call them “false”, because the monitors see what they see, nothing “false” about that. And one has to consider that the score is not only about the server, but it is reflecting the relationship between the monitors, and the server. As an incomplete representation of what “normal” clients would see. Which is what the monitoring system is all about. Not about getting best grades for the server itself.
To some degree, the system already does what you suggest, by giving preference to monitors that are closer (RTT-wise) to the server. The only thing it does not do is that it does not ignore servers that are farther away (it’s been suggested even before the new monitoring system was put in place to give less prominence in the graphs to severe outliers).
And from a practical point of view, it wouldn’t work, because probably the majority of countries/zones don’t have any monitors in them. Even continent zones sometimes have less than the currently required number of monitors (7) in them.
So from the comfortable position of a server operator in Europe or North America (and some select other places elsewhere), sure, one could be more “strict” (different aspects of “strictness” have been discussed before, e.g., tightening the performance requirements on servers). However, if the Pool still aspires to be a truly global project, then I don’t think further reinforcing an already existing imbalance/bias (with many causes) even further is the way to go.
This is not true, monitors that often give good results are selected,
It does not matter where they are.
A monitor that keeps testing good will made active.
Sorry, what you say doesn’t matter, the ping-time has no impact.
Monitors check your time, and if you are online…if they keep giving that as good, they are selected…nothing more to it.
Monitors that can not verify your correct time are not assigned. Simple as that.
Yes, that is the first preference, which takes precedence. But second is RTT. With mechanisms in place to avoid switching active and testing monitors too often, so there may be situations where testing monitors may have better values in one or both categories than active ones.
Sorry to say, but it does. Just compare the typical RTT values of active, testing, and candidate monitors. There may be exceptions, but you should be able to see a pattern.
And yes, score has first priority, but since the proposal was to consider some metric roughly related to RTT (country/zone), that is what I focused on, omitting the aspect about the scores for brevity and clarity (which apparently did not work as intended…).
Here’s an example for a server in India. Do you see a pattern (with exceptions) regarding where the monitors are located (as far as the country code in their name can be trusted), and regarding RTT values (considering that monitors in each section are first ordered according to score, but among monitors with same score secondly based on RTT, i.e., hkhkg1-3strqkc’s RTT is the lowest among Candidate monitors, and twtpe2-2kg9ezv’s RTT already exceeds 100ms)?
Here an example of a server in Japan which had some issues very recently, thus the varying scores from the monitors lending for some diversity. But again, do you see some sort of a pattern here regarding monitor location (with one exception, with LA and the West Coast of the USA typically well connected towards Asia), and regarding RTT?
Ok, one example only, could be an outlier. But look at as many servers as you like, and you should be able to discern a pattern.
I don’t even think this would help in your case. Looking at the full set of monitor scores for the server you indicated, it seems your server is in the USA, and the offending monitors with negative scores are as well. Thus, limiting monitors to the same country or zone as the server would not get rid of those offending monitors for you.
Nice graphs by the way! 
I agree here, but before the change the monitor (and beta monitors) where just unable to reach the system in time and your system was taken out of the pool because of that.
Monitors that do this too much will be replaced with others that are acting better.
As such the RTT only matters, it should not matter for scoring a system.
Also, it’s not just 1 monitor that can bring your score down, a bad scoring monitor will be taken out before anything drastic happens like your server(s) being taken out of the pool.
That was the entire point of the new system, to prevent that from happening.
I do not see that happening with the servers in the above samples.
I have the impression two, or even more (related but separate) things are getting confused here:
I don’t think that was the “entire” point of the new system, not least because already the previous system (with a closed group of monitor hosts) was doing it that way in my understanding, so not really much change there. Not sure whether it was explicit then already, and just not highlighted as it is now with the RTT being shown in a server’s monitor table, but I well remember that over time, at least my servers would predominantly be scored by monitors “close” to them geographically.
I think the main (but probably not only) points were to counter the attrition in monitors, and to provide more diversity regarding monitor locations. I believe already the previous system was considering the RTT as one factor when selecting monitors for a server, as it has become clear that while high RTT does not necessarily impact the metrics that ultimately determine the score for a server, the likelihood that it does increases as the RTT increases, as mentioned above. Besides other factors not necessarily directly related with RTT potentially impacting the score, e.g., non-negligible parts of the Internet in India apparently having asymmetric paths to the rest of the world causing remote monitors to see higher apparent offsets than when scored by local monitors.
No, but that also was not the point of the original poster in my understanding. Rather, that for reasons out of scope for this comment, some unspecified uneasiness about what was called “false negatives” lead to the desire to not have them considered for a server (whatever that means, as as you rightly point out as well, given the ample amount of monitors that have good scores for the server, the “false negatives” do not play any role whatsoever in determining the overall score of the server, and regardless of that in that particular example, the proposed mitigation would not even have helped).
