This is somewhere in between a blog post and a post asking questions. I had a hard time finding concrete documentation on setting up rsntp in front of regular ntpd, so I’m trying to share my setup—but it also doesn’t work flawlessly, and perhaps someone here has the clues.
I have a server at OVH (in their Canada data center), which sits in the pool dual-stack. (See its IPv4 and IPv6 pool listings if you’d like.) The IPv4 entry is also in the Brazil pool, which is pretty underserved, and thus the server sees a lot of traffic.
It’s running CentOS 7, is a(n older) quad-core Xeon, and has a 100Mbps unmetered network drop.
For reasons I don’t entirely understand, when I bump the IPv4 listing from the 25 Mbps pool setting to 50 Mbps, its score in the monitoring system starts to drop, and it periodically falls out of the pool. Server metrics look fine: even at peaks, ntpd doesn’t use more than about 3% CPU.
One thing I did notice is that running
ntpdc -c iostats shows thousands of dropped packets, though they’re a relatively small percentage. However,
ntpdc -c sysstat shows a signficantly higher percentage for “rate exceeded” (I do have some basic limiting configured.) So it seems like packets are getting dropped by ntpd for some other reason.
I could/should go to the source and try to figure out exactly where that can happen, but I took it as an excuse to play with rsntp. (Also: I recently rebooted the box to apply some kernel updates, and then went right into rsntp setup, so there are currently no dropped packets showing.)
I discovered it on this forum, but for those who haven’t seen it, rsntp is a multi-threaded NTP server written in Rust, which I’d almost liken to a cache. It sits in front of a “real” NTP server and bases most of its data on that, but does use real timestamps so its responses are good. The readme for the project shows it saturating a 1Gb network port with four cores.
Installation (CentOS 7)
With EPEL already set up,
yum install rust cargo gave me all I needed for a Rust runtime and compiler.
I then cloned the git repo for rsntp, and ran
make release in the checkout. That gave me an
rsntp binary in target/release. For now I’m just running it in screen while I tinker, which is obviously not a good long-term solution
By default, ntpd binds to *:123, which (1) is not where rsntp expects it to be, and (2) prevents rsntp from binding to port 123.
Frustratingly, ntpd doesn’t seem to present any means to change the port it listens on. In another thread on here, there was a recommendation to make ntpd listen on localhost only. For some reason, that configuration on my setup was overly restrictive, and NTP was unable to query any timeservers. (It’s as if it was locked down to only being able to use the loopback device, rather than just listening there.)
Since the overwhelming majority of my traffic is IPv4 and it’s a dual-stack box, I ended up compromising and going for this:
interface ignore wildcard interface listen ipv6
The first line keeps it from binding to everything, and the latter makes it bind to IPv6 addresses only. (For reasons I don’t fully understand, it still listens on 127.0.0.1 as well, but that’s perfect for my needs.)
The problem with this is that now I can only use IPv6 servers. There are some good nearby IPv6 servers so it’s not a big deal, but I feel like the configuration isn’t quite what I expected.
pointing rsntp at ntpd
My initial plan, with ntpd listening on IPv6, was to point rsntp at the IPv6 address. Unfortunately, for some reason, Rust doesn’t seem to like this:
[root@ns507230 release]# ./rsntp -4 3 -6 0 -a 126.96.36.199:123 -s [2607:5300:60:3308::1]:123 Server thread #1 started Server thread #2 started Server thread #3 started thread 'main' panicked at 'Client failed to send packet: Address family not supported by protocol (os error 97)', src/main.rs:369:27 note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
I suspect this is some oddity with the Rust runtime or something. (FWIW, I have 188.8.131.52.el7.) Using
[::1] as an IP was not any better.
So I ended up just pointing it at localhost, with
./rsntp -4 3 -6 0 -a 184.108.40.206:123 -s 127.0.0.1:123. And now it works.
By the way, the only reason for using 3 threads is that my fingers didn’t seem to want to type
-4 4 and I didn’t catch it until it was already running.
(Some of these are things I’m just interested in researching when I have time; not necessarily questions I expect the group to answer.)
- Under what circumstances does ntpd drop packets? Is a non-zero value indicative of a problem, or could it just be junk queries or the like?
- Does the Rust package I have just not support IPv6?!
- Is there a way to have ntpd bind to localhost-only without sacrificing the ability to connect to non-localhost timeservers? (Similarly, can I make it bind to IPv6 addresses only, but still query IPv4 servers?)
- How do people monitor rsntp? I’d love to get some metrics but it doesn’t seem to expose any. (Maybe I need to roll up my sleeves and learn some Rust!)
- (To be seen) Were my problems getting past 25 Mbps (pool setting) something that will be solved by rsntp, or was it something like a network issue?