Hi everyone,
The beta system now has a verification feature ready for testing. Please try it out! It doesn’t do much yet other than set a flag in the database, but assuming it’s reasonable to work with it can go on the production site and start helping to make sure only the proper server operators are adding servers (and slow down people adding random servers).
The UI is a bit subtle (for now). To start the verification process, click the “x” next to the server IP.
Feedback welcome.
My company’s forward web proxy is maping all internal IP addresses (including the NTP servers) of the outgoing HTTP/HTTPS connections onto one single IP address. That invalidates the option to verify the NTP servers I am maintaining via outgoing HTTP(S) access. Are you planning to implement different validation methods in addtion?
(EDIT: and there is no inbound HTTP/HTTPS access permitted by default. May be some method intrinsic to the NTP protocol?)
For stratum 1 server, we can set the refid to some random string decided by manage server.
server 127.127.28.1 prefer
fudge 127.127.28.1 refid XkCd
After validation this artificial refid can be removed. However I am not sure if this is possible for servers with stratum >1… And for devices like the LeoNTP, this tweak might be impossible.
Great! And what about startum 2, 3, … servers? Could you propose something?
Yes, but I need some help to figure out what would work! Ideally we can figure out what works for most and then have an exception process of semi-manual verification for the rest.
I didn’t implement the other ideas (yet) because I wanted to find out what will actually help in cases where the straightforward HTTP process doesn’t work.
One of my plans was to have a “telnet” type API similar to what I did with HTTP, but I don’t think that’d help in your case.
The refid thing is the only thing I could think of for doing verification over NTP, and it’s also limited and a bit awkward.
Throwing some ideas around:
The verification with a string put into a public readable place also only has to take place once I think? So having this changed temporarily should be workable even if you have to ask someone else in your organization to do it.
I think having multiple paths to verification is necessary since there will probably always be some edge case or another.
This might not be practical to implement on your end, but what about configuring NTP auth with a particular key with a monitoring server?
I think that it’s a sensible way to verify that the server operator has a server under his control. However, ideally, the verification should work just from the CLI, rather than having to click a button from a browser. It should suffice to access the verification link from the server’s IP address. Requiring that the server would additionally run an HTTP service or manually adding a DNS record or an NTP string would be overkill. And, no, only TLS accesses should be allowed.
It should become mandatory, should this method prove to be sufficient. As is, the pool is a cheap way to enable a vandal to start a low-level DDoS.
At least for ntpd, another approach would be to set a non-default variable with a verification string that can be retrieved by ntpq (mode 6 NTP).
C:\ntp>nq -c ":config setvar pool.ntp.org = VerifyCodeHere"
Config Succeeded
C:\ntp>ntpq -c "rv 0 pool.ntp.org"
pool.ntp.org=VerifyCodeHere
C:\ntp>Note that nq is a script which invokes ntpq with authentication options providing a keytype, control keyid (e.g. 123) and passwd. The key ID is configured in ntp.conf with both trustedkey 123 and controlkey 123 and is listed in ntp.keys with an algorithm and password, e.g.
123 SHA1 MyRuntimeConfigPassHowever, that’s just to illustrate the configuration change can be done without restarting ntpd. One could simply add to ntp.conf:
setvar pool.ntp.org = VerifyCodeHereand then restart ntpd.
Note there is a widespread misconception that mode 6 is somehow dangerous to allow remotely, hence lots of example configurations have something like:
restrict default noquery limited kod
restrict 127.0.0.1
restrict ::1which would prevent the verification ntpq query from being fetched remotely. This is likely due to confusion about the 2014 NTP amplification attacks using ntpdc -c monlist which is mode 7 and support for which was removed from ntpd in 2010 in ntp-dev an a couple of years later when ntp-stable finally was released with that change. It’s sad that so few NTP servers today allow ntpq queries due to this confusion.
At any rate, that issue can be worked around by publicizing the IP address(es) from which the verification query will come and asking those using this method to allow ntpq queries with loosened ntp.conf restrictions:
restrict 146.75.29.55 limited kod
restrict 2a04:4e42:77::311 limited kodThat is, whatever the default restrictions are minus noquery. Even better would be to remove noquery from the defaults, but I fear that bit of common wisdom is too far entrenched.
To be fair, there have been a number of security patches to ntpd over the years where the announcement advised using noquery as a mitigation and/or as a best practice. As a mitigation until ntpd is updated, I have no qualms, but I do wish it had not be recommended as a best practice because from my perspective that was deflection from the responsibility for ntpd to be secure even if the public is allowed to query using ntpq.
First: Verification works 
The curl / wget request must come from the ntp server address, correct ?
All the above verification methods are working with “software” ntp server but there is a small amount of hardware ntp server.
As optional other verification method you could temporarily change the PTR of the IP to a “specific” one wanted by the system.
Hi everyone,
First of all thanks to those of you who tried it out!
It looks like everyone who got a token also succeeded completing the process.
The goal is to have all new servers verified before joining the pool; and some of you with unusual accounts as well if not everyone.
@NTPMan Interesting situation with the outgoing connections on the network with the NTP server all going through a proxy. That’s a good scenario to make sure there’s a solution for; even if it’s a manual out-of-band verification process.
@alica, @mnordhoff, & @davehart Those are all interesting ideas on doing the verification through NTP, which could solve for example the situation @NTPMan mentioned. I considered some variations of this, but my thought was that in most situations where NTP configuration changes like this are possible, it’d also be possible to make an outbound TCP connection. I didn’t know that ntpd supports setting arbitrary variables for mode 6! Some of this could maybe solve @NTPMan’s situation, but it feels a bit obscure if that’s the only situation it helps.
@ebahapo I did consider having you get a token from the website and then submitting the token from the NTP server (basically the reverse of how I set it up), but I thought it more important to make the request from the NTP server as simple as possible. Requiring TLS requests would validate you are talking to the right validation server; but I can’t think of how a MITM attack would make sense since what we are testing really is just the three way TCP handshake. Not requiring TLS was to minimize software requirements on the NTP server; you can do a request with netcat or telnet if that’s what you have.
@Sebhoster The ACME-type validation are more geared towards validating a domain. For validating an IP it’d effectively just be “can I put a token on the server” which I think ends up being the same security as what I implemented (“can I fetch a token from the server”), without requiring an HTTP implementation on the NTP server. Happy to be convinced otherwise if I missed something. I agree on it being okay to just do this once; and that we likely need multiple paths to do it.
@apuls Yes, correct – the HTTP request has to come from the NTP server IP. PTR records would be a good proxy for “this is my IP space”, but my intuition is that they are too poorly maintained to be viable (and often server operators won’t have access to change them).
For the “hardware NTP” servers the most obvious workaround I thought of so far is to allow a request to come from a “nearby” IP (same /24 or /64?) as an “indirect verification” (maybe with a pool admin approving). I didn’t implement this yet as I wanted to see what other ideas and issues would come up first.
Thank you all for the input, suggestions and encouragement. If any of you try verifying one of your NTP servers and can’t, I’d love to hear about it so we can figure out what additional methods would solve it.
While mode 6 can’t be used for amplification attacks, it still exposes the list of recent clients to unauthenticated remote parties which is a privacy violation. While this can be avoided with further configuration i would refrain from forcing server operators to have to open their system up more than necessary.
It still remains viable for cases like the one @NTPman described. Having a pure ntp solution available to a ntp-related problem is definitely a good fallback, since that should work on all pool servers.
To put a token on the server you would need administrative access, while all users on the server can fetch one from there. The same goes for servers behind NAT - all devices and users behind the NAT are currently able to add an NTP server located at the same public IP address.
The question here is how “bomb-proof” the process should be versus how easy it should be for legitimate server operators. The process currently available on the test instance is simple, quick, and should work for most servers, but has those small holes (while I have to admit - they are mostly theoretical).
I’ve successfully validated one of my servers, but my first attempt failed. The server has several IP addresses and Wget chose the wrong one for the outgoing connection. So you might want include an additional command-line option for wget:
wget --bind-address=198.51.100.63 -O- https://validate4.ntppool.dev/b/
Replace 198.51.100.63 with the NTP server IP that you’re trying to validate. I can’t immediately find an equivalent option for cURL.
for curl it’s:
curl --interface 198.51.100.3 https://validate4.ntppool.dev/b/
Yes, but they have to be able to port forward UDP 123 (possibly with UPnP) so it has to be a relatively open network or the user must have admin access to the router, unless there’s already an NTP server on the router or a default port forward host, sometimes referred to as being a DMZ host. I don’t think we need to go that far into the weeds here.
It’s not a misconception. An amplification factor of 10 is still useful for DoS attacks. There might be other protocols with worse amplification available to the attackers, but that doesn’t mean mode 6 is ok.
“Might” is mighty misleading. There clearly are other more widely used protocols with much higher amplification factors that aren’t going away anytime soon. I agree amplification is a problem, but are you aware of anyone actually using NTP mode 6 for abusive amplification? I’m having a hard time understanding why it would be appealing with other more powerful mechanisms much more widely available.
Having a security issue is not rectified by someone else doing worse. That amplification attacks with NTP mode 6 are not widely used has probably more to do with the inclusion of noquery as default and best practice configuration than with the availability of other protocols. There is even an RFC acknowledging mode 6 as potential attack vector and recommending blocking mode 6 messages from outside your organization.
That being said, I still agree that temporarily allowing mode 6 messages sent from the IP of the verifying server would be ok to enable a verification mechanism like you described earlier.
I dispute the characterization of mode 6 as a security issue. It’s an information disclosure issue, but it doesn’t provide any access unless there is a bug in the implementation in the daemon. It’s a very minor amplification factor, but not enough to be attractive to those interested in UDP amplification because there are much more widely available and higher amplification factor services. It has been suggested that mode 6 queries should be at least as large as their responses to eliminate the amplification possibility, and I’m open to that change. So far, I have not been able to convince Harlan Stenn it’s a good idea.
The RFC does not identify mode 6 as a “potential attack vector” but only as an information disclosure risk.