Based on https://community.ntppool.org/t/stratum-pool-servers/1529/7 I thought I would ask the question 
Is there an “easy” way, other than brute force* to, on an ongoing basis, find pool servers that have a score between 5 and 15? (Following down the path of improving the quality of the pool)
Brute force = Do lots of DNS queries to find the pool server ip addresses then query the scores api endpoint to see and record what the current score is for each ip.
I can’t say I’d suggest going this route, but in addition to making tons of DNS queries it’s technically possible to use the server’s numeric ID. For example, https://www.ntppool.org/s/64166 redirects to https://www.ntppool.org/scores/45.115.225.48. You may have seen such shorter URLs in “Problems with your NTP server” emails.
This is still a fairly brute force method. On the other hand, this method would also find servers with a score <10, which you wouldn’t find via DNS queries.
@gombadi The scoring data still gets exported to Google BigQuery; and it’s also stored in avro files (though I don’t think they are accessible anymore; nobody was using them so I didn’t setup access again in one of the migrations over the years).
If you can tell what you are trying to do and it’s related to making the system better then I’m happy to help provide more access to the data.
Over the past while I have been playing around with a pool monitoring system. It gathers results that I have a look at. The results are public if anyone wants to have a look. This is a summary page and the format/data can change at any time though so you should not rely on the page staying the same.
While looking at the results I saw that some servers were always showing up in some results. Doing a manual score check of them I saw that quite a few had scores that never seemed (I only had snapshots in time) to reach 20 or even go above 15. They would quite often bounce around going in and out of the pool. For example https://www.ntppool.org/scores/23.106.249.200 has a score of 12.2 with most monitors detecting a clock offset of about 80mSec. Dear server owner can we help in improving the time that your server is giving out to users. Also https://www.ntppool.org/scores/43.229.16.158 currently with a score of 18.1 but looking at the test results for 20th Sept -
$ zcat *.gz | grep -v Tested_CAC | grep 43.229.16.158 | awk -F, '{print $2}' | sort | uniq -c | sort -rn
52 Tested_ERR
42 Tested_OK
More tests failed than worked. Can we work with the server owner to improve the performance of the server.
Instead of trying to monitor/check all the servers in the pool I thought it would be better to focus on servers with scores mostly between 5 and 15. I could make numerous requests to the pool api to check a servers current score but that seemed to much of a brute force attack so started this thread to see if there was a better way. There may be
That is pretty much what I am doing. Happy to answer any questions anyone may have.
The “constantly 80ms off” is pretty amazing. I wonder, too, how often that happens.
I’m curious too what the impact would be if we just tightened the accuracy requirement (I already did sometime in the last year or two); or if we should have some special logic where a server get a lower score if the time is off and not getting better.
Another area of inquiry would be what signals (all monitors saying “nope” in a particular way, for example) are strong signals that a server should be removed from the pool faster.
It looks like 23.106.249.200 has been advertising stratum 3, with a reference ID of 127.0.0.1 all year. Typically 127.0.0.1 means that the server is syncing to an internal clock. The root delay is 240 msec, which is huge. Last year the same server had a reference ID of 130.133.1.10 time.fu-berlin.de, at least part of the time, with a root delay of 220 msec.
My guess is that the operator has overridden the reference ID for some reason, and is still syncing halfway around the world. If true, its no wonder that there is an 80 msec offset.
There is quite a bit of asymmetry on IP networks. It can be difficult to distinguish server offset from monitor-server asymmetry.
Doesn’t have to be the case. As then it would list as Stratum 1 not 3.
It’s probably using the internal clocksource but synching with stratum 2 servers.
Because it’s on www.leaseweb.com, I suspect it to be a VPS, maybe on a host at very low priority.
I have seen many hosters to oversell virtual-machines and they often have very poor times, too little CPU-cycles to do proper work.
My 2cts.
Hey,
It’s Marcus from Leaseweb here.
I got a heads-up about this thread and the issues from Joachim, thanks for that!
Let me explain what happened here and how we’ve solved it.
We’re running multiple servers for the pool, all on dedicated Dell R240s. Some years ago there was a request on the mailing list for providers to join the CN pool with their servers in order to get it stabilized again. As we had the network capacity to spare, we did. Unfortunately, that took quite a beating on the single-threaded ntpd process we ran back then.
To improve performance, the team switched from ntpd to a multi-instance chronyd setup, in which one instance connects to the upstream time sources and multiple downstream instances are reachable for clients. During this configuration change, the copy flag in the internal server statement was missed, which caused the reference ID of the downstream instances to show up as 127.0.0.1 instead of the real reference.
This has been corrected, and all our pool servers should present a correct reference ID again. Unfortunately, this did not explain the 80 ms offset of this particular server.
However, during the config review of the current setup, we also decided to clean up our list of upstream servers to only include stratum 1 servers. This change did correct the offset, which makes me suspect it was indeed network related towards the selected upstream server.
In the meantime, all servers are churning away again, and apart from the ones in the CN pool, we have upped the net speed setting in the pool to 3 Gbit for both IPv4 and IPv6.
Additionally, we have set up monitoring which queries the pool API and alerts us when a server gets a less than perfect score (alongside the already existing process/stratum checks).
Hi Marcus
Thanks for your work looking into the issue and improving the quality of the pool.
As a followup to this thread I received an email from @ask with some options to get the scores. At the moment I am pulling the scores of known servers every 8 hours and looking at the last weeks results I see the counts, with ip addresses, below for servers with scores between 5 and 15.
Three times per day * 7 days gives just over 20 readings so any count approaching 20 suggests the server needs attention. Any count over 10 suggests that the server is spending at least half its time with a score between 5 and 15.
Pool servers spending the most time between a score of 5 and 15 in the last 7 days.
19 172.235.18.237
16 2600:3c08::2000:9bff:fe44:8ddb
16 2400:8905::2000:27ff:fe9b:4c1a
16 172.233.91.137
16 168.144.182.136
16 143.110.241.116
15 5.57.11.195
15 45.77.249.167
15 45.252.250.189
14 103.221.223.143
11 5.223.54.153
11 240e:982:2000:c::371:317
11 198.211.103.209
11 168.144.248.96
11 152.42.247.214
11 109.61.109.140
10 2603:c020:0:8369:0:ba11:ba11:ba11
10 151.236.20.238
9 2a14:67c1:b590:31ad::1
8 160.22.74.161
Happy to help any operators of the above systems to improve the quality of their systems.
Most of those server are in underserved zones, and are showing typical signs of overload at “netspeed” values that would be no issue at all in other zones.
So while the operators of those servers could lower their respective “netspeed” settings, the real solution to this kind of issue would be to address the topic of underserved zones.