Growing requests to http://north-america.pool.ntp.org/ from Android clients

@ahferroin7: I can see some Android 4.x in my log files. And also, the Dalvik version number seems to be the correct one for these Android versions.

At the moment I can see some clients constantly requesting my pool server with those requests:

GET / HTTP/1.1
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

All requests originate from different IP addresses, but all from the Comcast network.
These requests get an HTTP 400 answer, since (I assume) the Host: header is missing but required for HTTP/1.1.

In the log file of today there are about more or less exact 40 or 80 or 120 requests logged, coming from several IP address within the range 98.223.99.0 - 98.223.231.255.
The whole range 98.192.0.0/10 is routed or used by Comcast: https://whois.arin.net/rest/net/NET-98-192-0-0-1
Outside that range there were 40 requests each from IP addresses inside 68.50.200.0 and 68.60.240.0.
The mind-melting oddity is: There are always multiples of 40 requests per IP address. Not 15 or 27. Always 40, 80 or 120 requests. And all requests share the EXACT same user agent.

@lordgurke Yeah, I’ve yet to see the Dalvik version mismatch, and the handful of build ID’s I was able to check over the weekend are correct relative to the Android and Dalvik version too, so it looks like it isn’t spoofing te UA string (though that doesn’t mean it’s legitimate traffic, for every intelligent malicious actor out there, there’s at least a dozen stupid ones).

I’ve also seen the same kind of apparently Mac originated traffic too. You’re right it’s a 400 because of the lack of the Host header. I would not be surprised if the User-Agent header is bogus, that’s a five and a half year old version of Google Chrome it’s claiming to be, but it’s also violating the protocol in a way that Chrome never has (unless there’s a plugin involved). I have not yet come up with a reasonable way to deal with these as I kind of want to avoid making it trivial for people to DoS my web server due to automated reactive blocking of the offending addresses.

Ugh, this is annoying. You’d think having some headers from HTTP would help track down who’s doing it, but it doesn’t look like it.

The website is on a CDN (generously provided by Fastly) for this reason. I don’t actually keep logs, but if I’m reading the stats they keep then there are a disproportionate number of redirects (say from / to /en/ ?). Regular users would also download CSS, JS, etc.

The site gets an average of about a 100 requests a second, with peaks of about 5000 on an per-minute average basis. That seems about right compared to what I remember before the site was on varnish (thousands of requests at the top of the hour).

I don’t really have a good suggestion for what to do about it. One “solution” would be to make HTTP time an explicit service we offer (on a different hostname obviously) …

Well, the given headers for the Android traffic appear to be the defaults you get from the HttpURLConnection class, so either the individual who wrote the app is either actively trying to evade identification, or they’re really lazy.

As far as the macOS stuff, I’d be willing to bet that the headers are bogus and designed to just look like it’s macOS. There’s obviously some form of automation going on there, but the reported Chrome version didn’t really support this kind of thing very well, and Chrome has always followed the HTTP specs pretty much to the letter, so I think at least the UA header is spoofed. At the point at which that is spoofed, it’s reasonably safe to assume everything else is suspect too, so I don’t think there’s really anything to be found looking at that.

1 Like

After a few days, the distribution of request IPs seems really global and “normal” to me.
I made some statistics with Wireshark over the requested Hosts and URLs (seriously, there’s nothing you can’t do with Wireshark!).
I’ve uploaded a summary file there:
https://maxderdepp.de/files/ntp-http-hosts-1.yaml

The server on which I’ve captured this, has IP 192.96.202.120 (“dns-e.wdc-us.hosts.301-moved.de.”), so requests to this IP addresses or this host are most likely intrusion attempts.

The host “north-america” is not the only one pool host name queried. There are plenty of queries towards, for example, vendor zones for Fedora, CEntOS or Debian and others.

EDIT: That’s the latest map of request distribution:

Huh, I’ve yet to see any for any of the vendor zones on my systems, but that may just mean they’ve never been listed under the vendor zones.

In my case though, I actually have real legitimate traffic to my two systems, so I may need to start more aggressively handling this traffic if it gets significantly higher, as much as I hate the prospect of just dropping them instead of replying.

Is there an “official” way to deal with such requests that hit our NTP servers that also happen to be web servers? Like does the NTP project recommend/suggest/condone what should be done with those requests? I just realized I’m getting a bunch of these too.

Old topic, I know, but…

pool.ntp.org: Join the NTP Pool! has instructions about this, including a sample Apache config.

1 Like

Though that was written before this thread happened. I’ve set my servers to return 404 instead of redirecting http://north-america.pool.ntp.org/ (but I still redirect other subdomains). Though I’m sure the CDN can handle anything thrown at it.

I totally forgot about this thread. I’ve long since disabled logfiles for the pool redirect, haven’t noticed any spikes or anything in apache threads…

I’ll try to remember to re-enable logs this evening and see how many of these are still hitting my server.

Kind of curious since I thought cell phones sync their time with the wireless network via some special protocol? Unless maybe this is from non-cellular capable android based devices (tablets, notebooks, IoT, and whatnot)…

EDIT - After enabling logs again, sure enough there’s still those Android clients hitting the page, but the request rate isn’t that bad, maybe 6-10/min. Most of the clients look to be cell phones, and pretty new ones too (well, at least within the past 3 years models or so). That really doesn’t make sense since cell phones should get their time via the cell towers they are constantly pinging, and even GPS (if turned on)… Querying a web page like that is a really round-about and inefficient (TCP) way if you want proper time, not to mention the data usage…

Some programmers just need to be slapped upside the head, then kicked out to the curb, then dragged down to the shore and tossed in the ocean…

Did some googling and it looks like there are a couple potential sources from where all the queries are coming from.

First, I came across numerous boot logs from people’s android based phones, looks like on boot the system tries to make a couple SNTP requests to “north-america.pool.ntp.org”.

Second, specifically the “GpsLocationProvider.java” which uses a config file ‘/system/etc/gps.conf’ apparently has a hard-coded NTP line for “north-america.pool.ntp.org”… This is probably the source of the HEAD requests. It seems the phones will make a time query in an effort to speed up their GPS TTF.

From the sample files I saw online, it looks like there is a list of all the pool continents, just with the others commented out. And of course every phone manufacturer is too lazy to customize it (they probably don’t even know it exists) so it of course defaults to one zone…

Apparently people in other parts of the world have discovered this to being one cause their GPS takes so long to lock and there are lots of threads of editing that file, but I believe you have to root your phone, so of course the subset of people that have changed it from the default is probably pretty small…

I’m gonna say this is probably an ‘upstream’ android issue since apparently all manufacturers seem to be making the same exact query…

Here’s a link to the GpsLocationProvider.java just CTRL-F and search NTP and enjoy the reading… Looks like they have hardcoded in a 4-hour time check interval.

1 Like

I’m doing largely the same myself, though I’m returning a 421 (‘Misdirected Request’ is technically more accurate than ‘Page Not Found’). I’ve also got mine set up to send an absolute barebones response (response line, Date header, and only the truly mandatory other headers, plus an empty body) using the Lua module for nginx, which has helped significantly here in terms of bandwidth usage.

1 Like

I doubt whatever code is making the query has full error checking, either it gets the header and a timestamp or it doesn’t…

I compiled a list of useragents that have made queries in the past day… Just about all Android, every make & model you can think of… Top one being ‘vivo 1606’ which is an Indian phone apparently, and ‘SM-G570M’ which is a Samsung Galaxy J5 Prime.

Here’s some of the top clients… I couldn’t upload the complete .txt file.

 20 Dalvik/2.1.0 (Linux; U; Android 6.0.1; CPH1701 Build/MMB29M)
 20 Dalvik/2.1.0 (Linux; U; Android 7.0; SM-G615F Build/NRD90M)
 21 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-G532M Build/MMB29T)
 21 Dalvik/2.1.0 (Linux; U; Android 6.0; vivo 1601 Build/MRA58K)
 21 Dalvik/2.1.0 (Linux; U; Android 8.1.0; Moto G (5) Plus Build/OPSS28.85-13-3)
 22 Dalvik/2.1.0 (Linux; U; Android 5.0.2; vivo Y51L Build/LRX22G)
 22 Dalvik/2.1.0 (Linux; U; Android 5.1; A1601 Build/LMY47I)
 24 Dalvik/2.1.0 (Linux; U; Android 5.1.1; vivo Y21L Build/LMY47V)
 24 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-A520F Build/R16NW)
 25 Dalvik/2.1.0 (Linux; U; Android 8.1.0; SM-G610M Build/M1AJQ)
 26 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-J600G Build/R16NW)
 26 Dalvik/2.1.0 (Linux; U; Android 8.1.0; Mi A1 MIUI/V9.6.8.0.ODHMIFE)
 28 Dalvik/2.1.0 (Linux; U; Android 6.0; LG-K430 Build/MRA58K)
 28 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-G935F Build/R16NW)
 29 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-J700F Build/MMB29K)
 29 Dalvik/2.1.0 (Linux; U; Android 8.1.0; CPH1803 Build/OPM1.171019.026)
 30 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-G532MT Build/MMB29T)
 31 Dalvik/2.1.0 (Linux; U; Android 8.1.0; Redmi Note 5 Pro MIUI/V10.0.4.0.OEIMIFH)
 33 Dalvik/2.1.0 (Linux; U; Android 7.0; LG-M250 Build/NRD90U)
 37 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-J500M Build/MMB29M)
 38 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-G950F Build/R16NW)
 45 Dalvik/2.1.0 (Linux; U; Android 5.1.1; SM-J200G Build/LMY47X)
 68 Dalvik/2.1.0 (Linux; U; Android 8.1.0; SM-J701F Build/M1AJQ)
 71 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-G570M Build/R16NW)
 72 Dalvik/2.1.0 (Linux; U; Android 6.0.1; vivo 1606 Build/MMB29M)

Oh, I know they don’t have any error checking beyond following redirects and spamming requests. The distinct error code helps me with log analysis, and one can always hope a developer testing things will notice it.

Yep, that sounds about consistent with what I’m seeing.

It’s too bad we can’t realistically get everyone involved in the pool to just start dropping all connections that fit this pattern. While it’s not technically against the rules of the pool, it’s borderline abusive, and the only way that seems to work to get crap like this noticed outside of the community it affects is to be disruptive.

If someone could poke the Android developers to just get a vendor prefix and route all of this traffic to that, that might help too.

Remember Google have their own time service… they should switch to use it instead.

Why use 421 instead of a permanent redirect as suggested by the pool docs? Using a 301 uses minimal bandwidth, and means that a well-behaved client will remember the redirect and won’t come back to your site again (at least during the active session).

The docs suggest a permanent redirect for pool.ntp.org, which I have in place. The requests are for north-america.pool.ntp.org, which is not recommended to be a redirect.

Additionally though, the requesters are not well behaved, they don’t appear to properly follow redirect, and will just happily keep spamming requests until they get one back with a valid Date header. Even if they did follow redirects, why should I tell them to spam the main site when all they care about is the Date header that I can easily provide? So, 4xx so that it has a Date header, and 421 specifically because it is a misdirected request.

The docs suggest redirecting pool.ntp.org, *.pool.ntp.org and *.ntppool.org. (I don’t think the last one is used, though.)

FWIW, I think I said this above, but I also excluded http://north-america.pool.ntp.org/ from my redirects. It’s a trade-off between potentially inconveniencing a small number of people, and saving bandwidth for the abusive clients and Fastly.

I’ve always liked using Response 418 for ill-behaved things… hehe…

On a more serious note, I’ve searched through the code on http://androidxref.com/ and it looks like any base OS stuff are using basic SNTP requests (which I scanned through the SNTP java code too), which is configured from the gps.conf file, and it looks like almost all the phone manufacturers that had XML files available on that site keep the default of NA… rolleyes There is the off-chance some phone manufacturers have changed the SNTP java code though because some ISPs will block port 123 (back from the reflection attack days). I honestly didn’t dive that deep into the rabbit hole.

Someone did a write-up about a year ago: https://haystack.mobi/wordpress/index.php/2017/02/08/looking-at-ntp-usage-by-android-apps/

Looks like numerous android apps are actually making their own NTP requests to particular servers/pools? Why are apps doing this, wouldn’t it make sense to use the phone’s time? If a person really wanted to spoof time (or anything for that matter) it wouldn’t be that difficult to setup a proxy in front of the phone when it is using wifi…

So anyways, the “HTTP” requests are probably coming from some app, not the core OS which is making the SNTP requests.

Which brings up the other possibility I’ve been looking at, which is only treating HEAD requests (which seems to be all that these clients are sending) this way, and just redirecting regular GET requests like the docs suggest.

1 Like