I tested from France Telecom - Orange network. A second monitoring point is from Switzerland, 2a00:7580:60:211:: network. It is the same result, many success and rarely no NTP reply received.
Excellent news, thanks a lot!
Does anyone have a server that has reached the maximum score from the sgsin1-1a6a7hp monitoring station?
My NTP server located in Virginia has top score on all monitoring stations except this particular station located in Singapore.
95% - 92% of the servers it monitors gives healthy queries. This is the beauty of the new monitoring system is those routing difference are now accounted for. Some connections are going to have better routes and healthy checks more often then others. However to answer your question yes I have servers that score a perfect 20 from there but then have many others that don’t.
1 Like
@Clock As Chris said, for most servers it’s working fine. It’s not a problem that it doesn’t work for some as long as the system figures out to make another set of monitors the “active” ones for that server.
I’m hoping data like this will be helpful in identifying network providers or IXes dropping NTP packets after I’ve added a traceroute feature to the monitoring system.
4 Likes
The truncated offset values in the legend should be fixed (or at least better) now. Thanks again for reporting it.
1 Like
Ask, could you please hide the identity of these monitors, and maybe even keep some hidden?
From this paper: https://www.ndss-symposium.org/wp-content/uploads/ndss2021_1A-2_24302_paper.pdf
Currently, the NTP pool employs a single monitor server,
whose IP can be easily inferred; when a new timeserver is
registered to the pool, the first NTP queries to that server are
made by the monitor. A natural defensive measure, then, is
to extend the pool’s monitoring infrastructure to contain many
monitors and attempting to keep the identities of these servers
hidden. We believe that this will indeed raise the bar for an
attacker and limit its ability to inflict harm. We point out,
however, that preserving the anonymity of “secret monitors”
might prove hard against strategic attackers, as such attackers
can periodically misreport times and keep track of the IPs of
the timeservers that queried them before their server scores
were decreased by the NTP pool.
1 Like
also, this paper from 2023 raise serios concerns about the new monitoring system:
1 Like
I suspect hiding monitors would make debugging routing issues much more difficult. Hidding only IP address can be easily circumvented.
The paper seems to be about 6 months old, it doesn’t describe the current monitoring system using median score. I think the attacker would need to control a majority of monitors.
I like some of the suggestions such as warning about significant increases in the measured delay and falling back to a trusted set of monitors if inconsistencies in reported values are detected.
1 Like
What are the concerns they raise about the new monitoring system? Everything described is about the old system. In some regards the systems are the same, but mostly the new system should be more resilient.
The monitoring server IPs in the new system aren’t explicitly published, but a motivated attacker could easily figure out what they are.
I didn’t read these specific two papers in detail yet, but there have been many over the years. I’m happy to take suggestions for improvements (or after a discussion pull requests on GitHub), but most of the past research along these lines seems like an academic write up of the XKCD (except I am in California and there are a handful of you helping out week to week and a few thousand people running servers).
But yes, a lot of this depends on us running the system and the NTP servers not screwing up. We all do the best we can and have systems in place to mitigate common mistakes and failures. That’s how internet infrastructure generally works, isn’t it? I’d love to learn from other low-budget, volunteer run global internet infrastructure how they do things differently.
One of the concerns brought up in one of the recent papers (I think one of the ones you linked, @giovane) is around a few NTP servers getting disproportionately many of the requests in some countries. Improving on that is one of the next projects on my list now that the new monitoring system is in place.
5 Likes
What are the concerns they raise about the new monitoring system? Everything described is about the old system. In some regards the systems are the same, but mostly the new system should be more resilient.
So Kwon23 claims that " the new monitoring system currently being tested with 13 monitoring servers) is vulnerable to attack" . However, when push comes to shove, section 7.3 shows that if an attacker control multiple monitoring servers (apparenlty some are run by volunteers?, says the paper), it can evict some legitimate NTP servers from the pool by tampering with the scores. (seems plausible but a bit far fetched IMO).
Section 8 covers mitigations – pretty much adding stratum1 servers to sync clocks of monitoring servers and evaluting the RTTs. Section 8.2 proposes a new socring algorithm, which I think you folks did, by having medians which are good in exclujding outliers. Another thing is 8.3 where they suggest that ‘Adding unpredictability in monitoring’ would help.
In short: your new system with median scores seems to deal better with those scores.
The monitoring server IPs in the new system aren’t explicitly published, but a motivated attacker could easily figure out what they are.
Indeed, one of the two papers stated that the first NTP packets you receive if you add a new server from the pools i from the monitoring servers.
I didn’t read these specific two papers in detail yet, but there have been many over the years. I’m happy to take suggestions for improvements (or after a discussion pull requests on GitHub), but most of the past research along these lines seems like an academic write up of the XKCD (except I am in California and there are a handful of you helping out week to week and a few thousand people running servers).
Thanks for the openess, and I understand your position: you folks have been doing a great and free service to the Internet community for over twenty years, get virtually zero credit, and, just like a goalkeeper in soccer, are mostly remember when things allegedly don’ t go well – sometimes from academics that have incentives to finding issues instead of fixing them.
That’s how internet infrastructure generally works, isn’t it? I’d love to learn from other low-budget, volunteer run global internet infrastructure how they do things differently.
The NTP pool deserves far more credit that what it gets. As a user (and working for an operator who contributes to the pool), thank you for you folks time and service.
I didn’t know how crucial you folks were. The NTP pool is wikipedia-like timekeeper of the Internet. Like, one of our NTP servers received 7.2 billion queries from 158M clients and 52k ASes in 24 hours – one single server listed in the pool. And there are 4k+ serves. Thanks for doing this in you folks free time.
One of the concerns brought up in one of the recent papers (I think one of the ones you linked, @giovane) is around a few NTP servers getting disproportionately many of the requests in some countries. Improving on that is one of the next projects on my list now that the new monitoring system is in place.
That was on another (unpublished) paper, and that would definitely help.
thanks
3 Likes
That’s great, thanks!
I can confirm that it’s fixed.
Regards,
Chris R.
Thanks for the pointer to the day in the life paper. I notice checking the current monitoring for any.time.nl that I see the unusual statement that it’s not active in the pool, only in monitoring. My curiosity begs me to wonder why out loud…
The usual reason for this is to stop people from adding well-known public servers to the pool which shouldn’t be added, e.g. ntp.ubuntu.com, time.windows.com, etc. Because there’s no way to validate ownership of an NTP server at a given IP address (maybe NTS could fix this?) it’s technically possible for people to add those sites to their own pool accounts.
Speaking of which, I find myself wondering if Fatih Unlu is an alias for one of the pool admins, or just an enterprising nominator of public NTP servers to the pool:
It’s active in the pool, but at the moment only with it’s IPv6 address.
Hey,
thanks for sharing this information.
I see a lot of company NTP servers there
(HE,Superonline,Apple,etc).
I have changed all servers in this account to “Monitoring Only” and have taken further measures.
@ask is working on something not to add such public systems, but I don’t know how far he is with it.
1 Like
I’m happy to help. I’m hopeful of helping bring up some more monitoring sites at a later date in Chicago and Freemont DCs.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.