I’ve seen various references here to ‘the monitoring API’ plus some examples. This would be quite useful to me - is there any kind of documentation anywhere about this?
The code (incl. documentation sources) mostly is in the monitor and ntppool GitHub repositories, with public discussions in a new forum category, some threads in other categories such as the Pool Development category, and status of the actual upgrade, and pointers to further info, on the status page and in the Announcements category of the forum.
2 Likes
In addition to everything @MagicNTP wrote, also an explanation of the monitoring states and some documentation on the log api.
1 Like
How much data mining is allowed with the API? I’m preparing a paper on NTP managment, reliability and importance and looking for some historic evaluation. The automized monitoring concept is quite unique and intriguing.
Welcome, and yes — data mining the API is fine! In particular if you’re willing to engage with the project and community longer term.
I’ve seen the threads with suggestions for improving monitoring scores and DNS zone building. The main constraint on those efforts (beyond my time to write and deploy code) is that very few people besides myself and Steven Sommars are investigating and thinking deeply about the monitoring data — connectivity and routing issues, time sync problems, scoring edge cases, etc. More sustained eyes and brains are very welcome.
One note from experience: “work on a paper” often ends up with a paper titled “Volunteer-run critical infrastructure could be better!” — which, sure. It gets more frustrating when authors clearly spent significant time reverse-engineering how things work rather than looking at the source code, the data, or just asking me or the community. So please do ask!
For querying the HTTP API: it’s backed by ClickHouse, so if you’re doing bulk queries, please keep concurrency to around 2 requests at a time. The ClickHouse data goes back to roughly 2019. With the new monitoring system it’s about 2 billion checks per year.
The data is also available in Google Cloud BigQuery and for download as Avro files in object store, going back to 2008 — currently about 500GB, growing ~15-20GB/month. For larger-scale analysis the Avro files are probably your best bet.
1 Like