Is there an “official” way to deal with such requests that hit our NTP servers that also happen to be web servers? Like does the NTP project recommend/suggest/condone what should be done with those requests? I just realized I’m getting a bunch of these too.
Old topic, I know, but…
pool.ntp.org: Join the NTP Pool! has instructions about this, including a sample Apache config.
1 Like
Though that was written before this thread happened. I’ve set my servers to return 404 instead of redirecting http://north-america.pool.ntp.org/ (but I still redirect other subdomains). Though I’m sure the CDN can handle anything thrown at it.
I totally forgot about this thread. I’ve long since disabled logfiles for the pool redirect, haven’t noticed any spikes or anything in apache threads…
I’ll try to remember to re-enable logs this evening and see how many of these are still hitting my server.
Kind of curious since I thought cell phones sync their time with the wireless network via some special protocol? Unless maybe this is from non-cellular capable android based devices (tablets, notebooks, IoT, and whatnot)…
EDIT - After enabling logs again, sure enough there’s still those Android clients hitting the page, but the request rate isn’t that bad, maybe 6-10/min. Most of the clients look to be cell phones, and pretty new ones too (well, at least within the past 3 years models or so). That really doesn’t make sense since cell phones should get their time via the cell towers they are constantly pinging, and even GPS (if turned on)… Querying a web page like that is a really round-about and inefficient (TCP) way if you want proper time, not to mention the data usage…
Some programmers just need to be slapped upside the head, then kicked out to the curb, then dragged down to the shore and tossed in the ocean…
Did some googling and it looks like there are a couple potential sources from where all the queries are coming from.
First, I came across numerous boot logs from people’s android based phones, looks like on boot the system tries to make a couple SNTP requests to “north-america.pool.ntp.org”.
Second, specifically the “GpsLocationProvider.java” which uses a config file ‘/system/etc/gps.conf’ apparently has a hard-coded NTP line for “north-america.pool.ntp.org”… This is probably the source of the HEAD requests. It seems the phones will make a time query in an effort to speed up their GPS TTF.
From the sample files I saw online, it looks like there is a list of all the pool continents, just with the others commented out. And of course every phone manufacturer is too lazy to customize it (they probably don’t even know it exists) so it of course defaults to one zone…
Apparently people in other parts of the world have discovered this to being one cause their GPS takes so long to lock and there are lots of threads of editing that file, but I believe you have to root your phone, so of course the subset of people that have changed it from the default is probably pretty small…
I’m gonna say this is probably an ‘upstream’ android issue since apparently all manufacturers seem to be making the same exact query…
Here’s a link to the GpsLocationProvider.java just CTRL-F and search NTP and enjoy the reading… Looks like they have hardcoded in a 4-hour time check interval.
1 Like
I’m doing largely the same myself, though I’m returning a 421 (‘Misdirected Request’ is technically more accurate than ‘Page Not Found’). I’ve also got mine set up to send an absolute barebones response (response line, Date header, and only the truly mandatory other headers, plus an empty body) using the Lua module for nginx, which has helped significantly here in terms of bandwidth usage.
1 Like
I doubt whatever code is making the query has full error checking, either it gets the header and a timestamp or it doesn’t…
I compiled a list of useragents that have made queries in the past day… Just about all Android, every make & model you can think of… Top one being ‘vivo 1606’ which is an Indian phone apparently, and ‘SM-G570M’ which is a Samsung Galaxy J5 Prime.
Here’s some of the top clients… I couldn’t upload the complete .txt file.
20 Dalvik/2.1.0 (Linux; U; Android 6.0.1; CPH1701 Build/MMB29M)
20 Dalvik/2.1.0 (Linux; U; Android 7.0; SM-G615F Build/NRD90M)
21 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-G532M Build/MMB29T)
21 Dalvik/2.1.0 (Linux; U; Android 6.0; vivo 1601 Build/MRA58K)
21 Dalvik/2.1.0 (Linux; U; Android 8.1.0; Moto G (5) Plus Build/OPSS28.85-13-3)
22 Dalvik/2.1.0 (Linux; U; Android 5.0.2; vivo Y51L Build/LRX22G)
22 Dalvik/2.1.0 (Linux; U; Android 5.1; A1601 Build/LMY47I)
24 Dalvik/2.1.0 (Linux; U; Android 5.1.1; vivo Y21L Build/LMY47V)
24 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-A520F Build/R16NW)
25 Dalvik/2.1.0 (Linux; U; Android 8.1.0; SM-G610M Build/M1AJQ)
26 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-J600G Build/R16NW)
26 Dalvik/2.1.0 (Linux; U; Android 8.1.0; Mi A1 MIUI/V9.6.8.0.ODHMIFE)
28 Dalvik/2.1.0 (Linux; U; Android 6.0; LG-K430 Build/MRA58K)
28 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-G935F Build/R16NW)
29 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-J700F Build/MMB29K)
29 Dalvik/2.1.0 (Linux; U; Android 8.1.0; CPH1803 Build/OPM1.171019.026)
30 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-G532MT Build/MMB29T)
31 Dalvik/2.1.0 (Linux; U; Android 8.1.0; Redmi Note 5 Pro MIUI/V10.0.4.0.OEIMIFH)
33 Dalvik/2.1.0 (Linux; U; Android 7.0; LG-M250 Build/NRD90U)
37 Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-J500M Build/MMB29M)
38 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-G950F Build/R16NW)
45 Dalvik/2.1.0 (Linux; U; Android 5.1.1; SM-J200G Build/LMY47X)
68 Dalvik/2.1.0 (Linux; U; Android 8.1.0; SM-J701F Build/M1AJQ)
71 Dalvik/2.1.0 (Linux; U; Android 8.0.0; SM-G570M Build/R16NW)
72 Dalvik/2.1.0 (Linux; U; Android 6.0.1; vivo 1606 Build/MMB29M)Oh, I know they don’t have any error checking beyond following redirects and spamming requests. The distinct error code helps me with log analysis, and one can always hope a developer testing things will notice it.
Yep, that sounds about consistent with what I’m seeing.
It’s too bad we can’t realistically get everyone involved in the pool to just start dropping all connections that fit this pattern. While it’s not technically against the rules of the pool, it’s borderline abusive, and the only way that seems to work to get crap like this noticed outside of the community it affects is to be disruptive.
If someone could poke the Android developers to just get a vendor prefix and route all of this traffic to that, that might help too.
Remember Google have their own time service… they should switch to use it instead.
Why use 421 instead of a permanent redirect as suggested by the pool docs? Using a 301 uses minimal bandwidth, and means that a well-behaved client will remember the redirect and won’t come back to your site again (at least during the active session).
The docs suggest a permanent redirect for pool.ntp.org, which I have in place. The requests are for north-america.pool.ntp.org, which is not recommended to be a redirect.
Additionally though, the requesters are not well behaved, they don’t appear to properly follow redirect, and will just happily keep spamming requests until they get one back with a valid Date header. Even if they did follow redirects, why should I tell them to spam the main site when all they care about is the Date header that I can easily provide? So, 4xx so that it has a Date header, and 421 specifically because it is a misdirected request.
The docs suggest redirecting pool.ntp.org, *.pool.ntp.org and *.ntppool.org. (I don’t think the last one is used, though.)
FWIW, I think I said this above, but I also excluded http://north-america.pool.ntp.org/ from my redirects. It’s a trade-off between potentially inconveniencing a small number of people, and saving bandwidth for the abusive clients and Fastly.
I’ve always liked using Response 418 for ill-behaved things… hehe…
On a more serious note, I’ve searched through the code on http://androidxref.com/ and it looks like any base OS stuff are using basic SNTP requests (which I scanned through the SNTP java code too), which is configured from the gps.conf file, and it looks like almost all the phone manufacturers that had XML files available on that site keep the default of NA… rolleyes There is the off-chance some phone manufacturers have changed the SNTP java code though because some ISPs will block port 123 (back from the reflection attack days). I honestly didn’t dive that deep into the rabbit hole.
Someone did a write-up about a year ago: https://haystack.mobi/wordpress/index.php/2017/02/08/looking-at-ntp-usage-by-android-apps/
Looks like numerous android apps are actually making their own NTP requests to particular servers/pools? Why are apps doing this, wouldn’t it make sense to use the phone’s time? If a person really wanted to spoof time (or anything for that matter) it wouldn’t be that difficult to setup a proxy in front of the phone when it is using wifi…
So anyways, the “HTTP” requests are probably coming from some app, not the core OS which is making the SNTP requests.
Which brings up the other possibility I’ve been looking at, which is only treating HEAD requests (which seems to be all that these clients are sending) this way, and just redirecting regular GET requests like the docs suggest.
1 Like
So I only have like a day and a half of http logs, but out of the 6,988 unique IPs (most were one-time, but a few had anywhere from 4-10 requests in a day) with a useragent matching Dalvik (there were maybe 30 more requests that were not mobile), here is the Top 20:
2849 India
1248 Brazil
340 Mexico
309 Russian Federation
288 United States
283 Argentina
148 Turkey
137 Peru
134 Malaysia
130 Saudi Arabia
108 Italy
105 Philippines
94 South Africa
88 Germany
84 Chile
60 Algeria
57 United Kingdom
56 Colombia
55 Taiwan, Province of China
41 Spain
So now the plot thickens… Not a single request from China!!! (Though there are plenty NTP requests)…
Just ran an update to see how the numbers are panning out… 12,279 requests total… So ~5.2k more since yesterday… Something is definitely up with India & Brazil… I don’t think the percentages will drastically change, I’m going to disable logging now for the redirects… Curious there still is not a single request from China… Overall this really isn’t that many requests when you consider NTP is serving up hundreds, or even thousands of requests per second…
4897 India
2133 Brazil
582 Mexico
504 Russian Federation
501 United States
471 Argentina
299 Turkey
236 Saudi Arabia
236 Malaysia
206 Indonesia
205 Peru
174 South Africa
169 Philippines
163 Italy
153 Germany
139 Chile
104 United Kingdom
102 Colombia
98 France
93 Algeria
83 Taiwan, Province of China
73 Spain
64 Korea, Republic of
63 Thailand
54 Japan
53 Hong Kong
51 Israel
49 Canada
33 Singapore
31 Nigeria
NTP also has significantly less overhead on the wire than this does.
The lack of stuff from China doesn’t surprise me much though, this is probably getting blocked by the great firewall.
What has me really interested is that this doesn’t seem to match up with population figures, even when scaled based on expected percentage of people in the respective countries with Android phones. I wonder if there’s some kind of buffering effect happening because of part of the north-america pool not running web servers?
The Android system NTP requests are fine (they have a vendor zone since many years ago, the NTP client appears well behaved). The HTTP requests are just wrong.
The NTP Pool doesn’t (at this time anyway) have a “time over HTTP” service, so those are just random bad requests.
Some apps tend to be more popular in some countries than in others. If memory serves, Viber is very popular particularly in India and Brazil, but it’s just an example.
Just FYI that nowadays we are receiving >20 requests per second from MIUI/Black Shark alone from China.
Social media link with a screenshot: https://twitter.com/felixonmars/status/1398442763331514368
1 Like