I see those requests on my US server, too.
I plotted the requests towards my system on a GeoIP map. Unfortunately, there is no real “hot spot”.
Until now, the device types (if you trust the User-Agent:-Header) are by 100% Android with many different makes and models. If it’s caused by an app: Why is there no traffic from iOS devices?
It’s either an Android exclusive app or the iOS variant is going other ways to get the current time.
I will start monitoring HTTP traffic towards “north-america.pool.ntp.org” by tomorrow (monday) in our office.
Maybe, someone is also using this specific app so we can find out, what is causing these requests.
@avij That’s interesting, I’ve not seen anything on IPv6 yet, and on top of that, I actually hadn’t seen any UA headers from any Android version before 5.1 either. I also find the presence of the Accept-Encoding header a bit odd, as gzip encoding a response will save absolutely zero time on a HEAD request unless there are a ton of headers being returned. As far as the traffic volume. it’s enough with the net speed set to 100MBit that it was swamping the kernel memory for TCP connections on my systems on occasion before I disabled keepalive. As far as configuration, I’m likely going to do similarly to what you’ve done, although I’m tempted to return 421, as that more accurately describes the situation, and will probably add some particularly draconian rate-limiting as well.
@lordgurke Thanks for the map! For what it’s worth, that’s actually a rather typical global distribution for a generic Android app, with the distribution itself suggesting very limited localization (my guess based on what’s shown is that it’s probably only one or two languages, most likely English and/or Spanish). The fact that they’re going for ‘north-america.pool.ntp.org’ combined with the heavy US and Mexico usage seems to indicate it was probably made in the US. As far as no iOS traffic, that doesn’t really surprise me either, as poorly thought-out stuff like this tends to be single-platform. Overall, it sounds like some college student wrote up a game and doesn’t trust the system time for some reason (or thinks he can’t trust the system time).
Clarification on IPv6: All request to north-america.pool.ntp.org are made via IPv4 because that DNS entry does not have an IPv6 address, but when I redirect that request to some other hostname that has both an IPv4 and IPv6 address, some of those requests are made via IPv6. If north-america.pool.ntp.org had an IPv6 address in DNS, there would be some requests to that address over IPv6 as well. This does not matter much, but I only wanted to point out that some of those clients do have IPv6 capability.
Somewhat ironically, I got another burst of traffic on one of my systems, and this seems to keep things in a reasonably sane state on the server end (no more than about 8 connections at a time from the offending devices, each one lasts about 1 second at most and uses almost no processing power (I think turning off gzip compression was what really changed that), and a 421 return code still sends the Date header. I’ve also explicitly called out a match for www.pool.ntp.org because believe it or not I’ve actually gotten a few stray requests with that in the host header.
Yup. That happened to me once a few years ago. And of course I redirected *.pool.ntp.org to http://www.pool.ntp.org/, so it caused an infinite redirect loop with the client until I changed the configuration…
@mnordhoff Good point. On the other hand, it’s almost certainly going to be a case of either a horribly misconfigured client or active malicious intent (or possibly DNS issues, but I doubt that that will be the case), so I may check that explicitly and return something else to flag the request as obviously bogus…
@ahferroin7: I can see some Android 4.x in my log files. And also, the Dalvik version number seems to be the correct one for these Android versions.
At the moment I can see some clients constantly requesting my pool server with those requests:
GET / HTTP/1.1
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
All requests originate from different IP addresses, but all from the Comcast network.
These requests get an HTTP 400 answer, since (I assume) the Host: header is missing but required for HTTP/1.1.
In the log file of today there are about more or less exact 40 or 80 or 120 requests logged, coming from several IP address within the range 98.223.99.0 - 98.223.231.255.
The whole range 98.192.0.0/10 is routed or used by Comcast: https://whois.arin.net/rest/net/NET-98-192-0-0-1
Outside that range there were 40 requests each from IP addresses inside 68.50.200.0 and 68.60.240.0.
The mind-melting oddity is: There are always multiples of 40 requests per IP address. Not 15 or 27. Always 40, 80 or 120 requests. And all requests share the EXACT same user agent.
@lordgurke Yeah, I’ve yet to see the Dalvik version mismatch, and the handful of build ID’s I was able to check over the weekend are correct relative to the Android and Dalvik version too, so it looks like it isn’t spoofing te UA string (though that doesn’t mean it’s legitimate traffic, for every intelligent malicious actor out there, there’s at least a dozen stupid ones).
I’ve also seen the same kind of apparently Mac originated traffic too. You’re right it’s a 400 because of the lack of the Host header. I would not be surprised if the User-Agent header is bogus, that’s a five and a half year old version of Google Chrome it’s claiming to be, but it’s also violating the protocol in a way that Chrome never has (unless there’s a plugin involved). I have not yet come up with a reasonable way to deal with these as I kind of want to avoid making it trivial for people to DoS my web server due to automated reactive blocking of the offending addresses.
Ugh, this is annoying. You’d think having some headers from HTTP would help track down who’s doing it, but it doesn’t look like it.
The website is on a CDN (generously provided by Fastly) for this reason. I don’t actually keep logs, but if I’m reading the stats they keep then there are a disproportionate number of redirects (say from / to /en/ ?). Regular users would also download CSS, JS, etc.
The site gets an average of about a 100 requests a second, with peaks of about 5000 on an per-minute average basis. That seems about right compared to what I remember before the site was on varnish (thousands of requests at the top of the hour).
I don’t really have a good suggestion for what to do about it. One “solution” would be to make HTTP time an explicit service we offer (on a different hostname obviously) …
Well, the given headers for the Android traffic appear to be the defaults you get from the HttpURLConnection class, so either the individual who wrote the app is either actively trying to evade identification, or they’re really lazy.
As far as the macOS stuff, I’d be willing to bet that the headers are bogus and designed to just look like it’s macOS. There’s obviously some form of automation going on there, but the reported Chrome version didn’t really support this kind of thing very well, and Chrome has always followed the HTTP specs pretty much to the letter, so I think at least the UA header is spoofed. At the point at which that is spoofed, it’s reasonably safe to assume everything else is suspect too, so I don’t think there’s really anything to be found looking at that.
After a few days, the distribution of request IPs seems really global and “normal” to me.
I made some statistics with Wireshark over the requested Hosts and URLs (seriously, there’s nothing you can’t do with Wireshark!).
I’ve uploaded a summary file there: https://maxderdepp.de/files/ntp-http-hosts-1.yaml
The server on which I’ve captured this, has IP 192.96.202.120 (“dns-e.wdc-us.hosts.301-moved.de.”), so requests to this IP addresses or this host are most likely intrusion attempts.
The host “north-america” is not the only one pool host name queried. There are plenty of queries towards, for example, vendor zones for Fedora, CEntOS or Debian and others.
EDIT: That’s the latest map of request distribution:
Huh, I’ve yet to see any for any of the vendor zones on my systems, but that may just mean they’ve never been listed under the vendor zones.
In my case though, I actually have real legitimate traffic to my two systems, so I may need to start more aggressively handling this traffic if it gets significantly higher, as much as I hate the prospect of just dropping them instead of replying.
Is there an “official” way to deal with such requests that hit our NTP servers that also happen to be web servers? Like does the NTP project recommend/suggest/condone what should be done with those requests? I just realized I’m getting a bunch of these too.
Though that was written before this thread happened. I’ve set my servers to return 404 instead of redirecting http://north-america.pool.ntp.org/ (but I still redirect other subdomains). Though I’m sure the CDN can handle anything thrown at it.
I totally forgot about this thread. I’ve long since disabled logfiles for the pool redirect, haven’t noticed any spikes or anything in apache threads…
I’ll try to remember to re-enable logs this evening and see how many of these are still hitting my server.
Kind of curious since I thought cell phones sync their time with the wireless network via some special protocol? Unless maybe this is from non-cellular capable android based devices (tablets, notebooks, IoT, and whatnot)…
EDIT - After enabling logs again, sure enough there’s still those Android clients hitting the page, but the request rate isn’t that bad, maybe 6-10/min. Most of the clients look to be cell phones, and pretty new ones too (well, at least within the past 3 years models or so). That really doesn’t make sense since cell phones should get their time via the cell towers they are constantly pinging, and even GPS (if turned on)… Querying a web page like that is a really round-about and inefficient (TCP) way if you want proper time, not to mention the data usage…
Some programmers just need to be slapped upside the head, then kicked out to the curb, then dragged down to the shore and tossed in the ocean…
Did some googling and it looks like there are a couple potential sources from where all the queries are coming from.
First, I came across numerous boot logs from people’s android based phones, looks like on boot the system tries to make a couple SNTP requests to “north-america.pool.ntp.org”.
Second, specifically the “GpsLocationProvider.java” which uses a config file ‘/system/etc/gps.conf’ apparently has a hard-coded NTP line for “north-america.pool.ntp.org”… This is probably the source of the HEAD requests. It seems the phones will make a time query in an effort to speed up their GPS TTF.
From the sample files I saw online, it looks like there is a list of all the pool continents, just with the others commented out. And of course every phone manufacturer is too lazy to customize it (they probably don’t even know it exists) so it of course defaults to one zone…
Apparently people in other parts of the world have discovered this to being one cause their GPS takes so long to lock and there are lots of threads of editing that file, but I believe you have to root your phone, so of course the subset of people that have changed it from the default is probably pretty small…
I’m gonna say this is probably an ‘upstream’ android issue since apparently all manufacturers seem to be making the same exact query…
Here’s a link to the GpsLocationProvider.java just CTRL-F and search NTP and enjoy the reading… Looks like they have hardcoded in a 4-hour time check interval.
I’m doing largely the same myself, though I’m returning a 421 (‘Misdirected Request’ is technically more accurate than ‘Page Not Found’). I’ve also got mine set up to send an absolute barebones response (response line, Date header, and only the truly mandatory other headers, plus an empty body) using the Lua module for nginx, which has helped significantly here in terms of bandwidth usage.
I doubt whatever code is making the query has full error checking, either it gets the header and a timestamp or it doesn’t…
I compiled a list of useragents that have made queries in the past day… Just about all Android, every make & model you can think of… Top one being ‘vivo 1606’ which is an Indian phone apparently, and ‘SM-G570M’ which is a Samsung Galaxy J5 Prime.
Here’s some of the top clients… I couldn’t upload the complete .txt file.
Oh, I know they don’t have any error checking beyond following redirects and spamming requests. The distinct error code helps me with log analysis, and one can always hope a developer testing things will notice it.
Yep, that sounds about consistent with what I’m seeing.
It’s too bad we can’t realistically get everyone involved in the pool to just start dropping all connections that fit this pattern. While it’s not technically against the rules of the pool, it’s borderline abusive, and the only way that seems to work to get crap like this noticed outside of the community it affects is to be disruptive.
If someone could poke the Android developers to just get a vendor prefix and route all of this traffic to that, that might help too.
