Monitoring and DNS services seems to have issues today ( 2026-03-01)

At inception, scale will be a challenge. In my opinion focusing on certain regions and limiting to IPv6-only might help.

Not anymore, because I’m close to retirement - but 5 years ago I wrote this.

1 Like

Cloudflare is having too much trouble with their backbone as we can see it the last 6 months, i would not centralize everything on them :slight_smile:

Why not contribute to the one we have? Some relatively approachable ways to help: reviewing PRs (and with some experience and trust built up, approving and merging them too), helping answer server operator emails, and working on the scoring system (it’s open source, the data format is simple, and all the past monitoring data is available to run simulations).

For example @ebahapo asked about the pool serving bad time; one could estimate this by looking at the monitoring data immediately after the system came back. Identify servers that had an acceptable score when the outage started and did not when monitoring was back online. This is tedious and time consuming, but not super difficult.

The pool has been working for 20+ years. George Bush was the US president when it started. YouTube was just getting started. For me the most important thing is that it keeps running for however many more decades it’s useful to the internet.

To that end: there’s a skeleton legal entity and I’ve been working on setting up an advisory board and eventually something more formal. The new clusters are hosted with help from two very different organizations in different parts of the world, each with different input, relevant experience and an understanding of how the NTP Pool is different from a time service run by any of the big internet companies of the decade.

To answer @marco.davids’s other questions: a small number of people have access to backend systems. The project doesn’t really have a budget beyond people’s time and small expenses. We have generous supporters for the central infrastructure, but the real cost is a huge amount of time. Getting more people involved in the day-to-day is the prerequisite for better governance, not the other way around.

To clarify the recent outages: we had a few different issues, but the majority was the graphs and website pieces. The underlying monitoring and DNS systems were working as expected (as @avij noted). I know that’s cold comfort when you’re trying to troubleshoot a low score and can’t see the data.

The infrastructure move has been a huge amount of work in my spare time after day job and personal life. Equinix Metal is sunsetting so there’s external pressure to move fast. The move is close to done, but I’m really tired.

On IPv6: this and the lack of participation in general is why I’m not enabling it until I have bandwidth to monitor it closely. Other than @stevesommars and myself I don’t think anyone is regularly looking at the quality of the pool.

We have a few volunteers who help answer emails from server operators, but most are still answered by @apuls and me. If you’ve been involved for a while and have time, there are lots of ways to help.

2 Likes

In the next month or so I will be retiring from a long time as a sysadmin / site reliability engineer. Very interested in finding a way to help out the pool. Have been a server operator for many years as well.

Is there somewhere I, and others, can indicate our interest and availability in helping out?

Not sure exactly what ā€œlooking at the quality of the pool.ā€œ relates to but having build a number of tools over the years I am looking forward to contributing what I can. Just need a pointer to what and how I can help with.

3 Likes

@ask I’m also willing to assist. Unfortunately i have no programming experience, but can look at aswering emails and looking at day-to-day issues.

2 Likes

If you could show me the best places to look or if it’s even possible for us to access the backend monitoring (only monitoring). I’d be happy to check in on it more often myself.

3 Likes

Thanks @gombadi, @Kets_One, and @Knot3n — really appreciate you all stepping up.

I want to give each of you a proper reply with specific pointers on where help would be most useful, so I’ll reach out individually over the weekend. In the meantime, thank you for the patience and the willingness to pitch in!

3 Likes

Just give a shout where you need help and i try to help out where ever i can !