It would surely help with debugging to get an example of such a monitor.
Just to confirm, are you aware that if you run NTP servers and NTP monitors, your own NTP servers won’t be monitored by your own monitors? This is by design.
3 Likes
Sure, but as long as we don’t have anyone with the competence and the time to delve into the innards of the code, educated guesses are all we have right now. And from own experience in many cases, those can go a long way in understanding something going on inside a blck box. Sometimes even better, when the system in the black box is so complex that the overall behavior cannot be derived from the code alone, kind of like not seeing the forest for all the trees. (In context of KI, there is a concept called explainability that may be lacking, but it can also happen in other complex systems.)
I don’t really understand why you are so reluctant to share those monitors here, but the reasons are entirely your own, don’t want to pry in any way. Just saying that even if you are not interested in pursuing this at this time, others still might be, and sure would appreciate that information.
E.g., checked with a newly created monitor, and kind of as I suspected, while it has not been accepted yet (i.e., not in global active state), it is monitoring way fewer servers, actually less than three dozen it seems (with IPv4 in this case). So that could be an educated guess as to what may be happening regarding those monitors as well. Knowing which ones those are, others could check other servers, and maybe a pattern emerges.
Had you read the whole thread with attention, you wouldn’t be making such a statement. Hint: Public monitor pages - #18 by ebahapo
Actually I did read (and reread a few times) that message but didn’t quite understand why you used an unnecessarily complex wording in that message.
But hey, if you do not want to share ANY of your information, it’s fine for me. If this continues to be a problem for you, feel free to contact server-owner-help@ntppool.org for private assistance. The other people on this community forum can continue discussing issues directly related to the topic of this thread, “Public monitor pages”.
1 Like
As for what could be shown on the public monitor pages, there are some lovely graphs in the other topic, especially the RTT vs Offset graph. Those graphs are generated from the point of view of a single NTP server, but those could just as well be generated from the point of view of a single monitoring node.
I did a tcpdump capturing outgoing ntp packets on my one monitor. After 10000 packets (taking 1 hour 15 minutes) there were 1605 unique addresses. After 40000 packets (taking 3 hours), there were 6647 unique addresses. So I think a monitor does monitor all ntp servers. Just very slowly for the ones that it is not in the active or testing state.
2 Likes
I agree with @john1, I saw this also…as the number of tests changed when it became active state.
You can also see this, as the green-dots are far more frequent then when you hover the mouse over the grey-dots.
Those dots are far less, maybe just 10% of the time of green-dots.
That would suggest candidates poll less, but are just scoring themselves over time to become active for a server, rather then providing usefull info.
My opinion.
If that were the case, when opening the scores page of any server in the pool and I’d find all of my monitors in it. Yet, I don’t. Do you?
A-ha, so this is about your monitors. Good good.
Piecing together these statements of yours:
“I know of two other monitors in my country. Yet none of my servers is monitored by any of them.”
“Those monitors have been added some time ago and monitor other servers in the country and abroad, but none of my servers.”
“If that were the case, when opening the scores page of any server in the pool and I’d find all of my monitors in it. Yet, I don’t.”
.. I get the impression that you are expecting to see your monitors shown on your NTP server score pages. Is this exposition correct? If not, please fill in the missing information.
2 Likes
I don’t see them on my own servers, but that is by design. I do see them on the ~10 that I checked. I checked a few from Europe, South and North America and a few South African ones that are not mine.
As I said above, there are other monitors in my country, two of the four, that have never visited my servers. I do find such monitors and my two monitors visiting other servers.
And, yes, I’m fully aware that one’s monitors don’t visit one’s servers. If anything because I see them as paused in the scoring page for my servers.
I found that they do. They are just not considered for the scoring, which in my understanding is what the “paused” is all about.
And yes, there are various inconsistencies that I saw in how the system behaves that I can’t really explain, among them similar cases as to what you describe. But just because there are some exceptions/corner cases, I don’t think that changes how the system behaves for the large majority of servers and monitors.
As we have server pages, may we have public monitor pages?
Fundamentally because I don’t want to incentivize unhelpful behavior from a monitor operator. We don’t want “good” results from each monitor, we just want … data. I didn’t figure out what to show that’d be most relevant.
As some of you pointed out, the system filters out servers from the same account, in the same IP network (and ASN? I forget). The intention is to maximize the accuracy from the perspective of an average (median?) NTP user.
The system tries to choose “active” monitors (those with higher frequency of checks) for each server based on those that are close and working well. The testing monitors are “backups” and the “candidates” are checked every so often to have extra “background data” (and to select the “testing” servers).
However, I’m pretty sure a bunch of the choices I made for timing and counts aren’t holding up to the large number (yay!) of monitors that have been setup. I’ve been working on moving the underlying database (for the whole system) to Postgres which I hope will clear up some database contention (and generally lessen the toil managing the database) so I haven’t looked too closely at it yet.
I am interested in specific ideas for what would be interesting to see for a monitor operator, beyond the basic counts. Again keep in mind that it’s important not to encourage comparison of which monitor is “better”. We want monitors across a diversity of locations and networks, not just the ones in “good” networks/locations.
3 Likes
I think probably maybe most likely this is just unlucky timing with when the monitors or servers were added and the algorithm currently gets “stuck” when things are good enough.
2 Likes
If not what exact servers being monitored, then maybe which zones it talks to most. Like, from snooping with tcpdump, I know mine’s been poking at some Russian servers, so I’d be interested to see just how often it does that, and what other zones it frequents.
Generally, as a volunteer, I just like getting a sense of how much I contribute. Like, on the server page, I can see that one of my servers get 496.19 ‱ of requests in Norway, which I interpret to mean that my 2 pool servers each get a ~5% share of the Norwegian traffic, from DNS perspective.
So for the monitor, I’d expect it to mainly monitor Norwegian servers, perhaps several Swedish, Danish, Finnish and Russian servers, and who knows where else. Would be fun to see some stats on that.
2 Likes
Not necessarily only for the monitor operator but for the community at large (and the point isn’t something that hasn’t been mentioned before, but from a different perspective):
- Which areas is a monitor covering, especially the Active ones?
- RTT vs. offset and perhaps packet loss.
- And certainly not specific to a particular monitor, but more a meta-evaluation: Which areas don’t have enough “local” monitor coverage, but are only covered from further away? Unlike the pool itself to a large extent, the monitoring system, or rather the amount and geographical distribution of monitors, is not about load sharing, so I don’t think that metrics like monitors per server would be helpful. Rather, a metric that reflects how well the monitoring system reflects what “local” actual clients would see would be more to the point in my view. Ideally, at least the majority of Active monitors for a server should be “local”, or at least “regional”. And the metric would show where the monitoring system is still further away from that ideal, and by how much, so as to highlight to volunteers where it would be worthwhile to set up additional monitors, or any monitors at all, to fill the gaps.
Something like “heat maps” would be interesting for the above.
ASN is not a good indicator, for the same ASN may be spread all over the world, as is the case of cloud services. IP network is though.
I think that is what would be most interesting to monitor operators, to get this sense of accomplishment.
For me it would be helpful to remove poor monitors (for me) on the manage-page so I can see only the results on the graph of monitors that score my ntp-servers well. As they are probably have the best results in accuracy, so I can see how my ticker is doing. To see how they judge me, all the rest is not important, may be better to hide the rest unless you need the extra info. Could be helpful if your NTP-server is consistently bad scored. To figure out why? By default too much information.
However, it would be helpful to score monitors globally, dividing in e.g. Asia / Europe / Africa / etc per continent how they work on average. So people have an understanding why e.g. many monitors don’t work for China to name a country. It may help understanding how good the peers are in a country. Also, it tells you why so many monitors can’t reach your ntp-server.
I mean, to see where your monitor often fails and where it works well. It would help seeing how well your connection goes in any direction.
As for scoring a monitor on it’s own, I agree, it’s simply useless and has no purpose.
I have seen comments where people take RTT into account, still don’t get it, that’s not the purpose of the monitor-system.
But if you run a monitor and it’s never selected because it has poor peering, that would help the monitor-operator to call his ISP and ask why it happens. Such scoring would help. For all I care, make it simple reached/not-reached number. You probably know better then me how to do this.
Other then that, I do not want to know if my monitor is better then any other, I just want to know if it does it’s job or not.
Methinks that I figured out why none of my servers are monitored by the other monitors in my home country: all my servers are IPv6 only and it’s possible that all the other monitors in my country are IPv4 only.
As I said before, perusing the status pages of other servers in my country, I’d see mine and other monitors in my country, but not of all. I finally noticed that my monitors are probably the only dual stack ones in my country. For in the status pages of IPv6 servers in my country I can only see my monitors, whereas in the status pages of IPv4 servers in my country I can see mine and other monitors in my country, which I don’t see monitoring my IPv6 servers.
That would make sense and confirm what others reported here, that a monitor likely visits all servers in the world.
And the algorithm is mostly correct, selecting monitors in the same or neighboring countries, as they’ll tend to reach the top score for a stable server more consistently due to shorter RTT.
2 Likes