Recent NTP pool traffic increase

Server operators
1

IPv4 Traffic started increasing on Dec 13. IPv6 traffic was unchanged (times in this graph are US/Central)

Traffic follows the pattern of peaks in the US Daytime/evening, valleys in the US nighttime. Traffic was seen to a wide variety of different regional NTP pools: New Zealand, United Kingdom, United States, Japan, etc.

Traffic is coming primarily from US networks: Cable, Cell Phone, and DSL/Other (times in this graph are UTC):

all-sources-traffic.png900Ă—600 13.1 KB

Reports of NTP traffic increases from NTP server operators and NTP in access networks have been in the range of “5x - 20x” normal levels.

The majority of traffic (>50%) has the following NTP flags set:

NTPv4, length 48
        Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 4s, precision -6
        Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
          Reference Timestamp:  0.000000000
          Originator Timestamp: 0.000000000
          Receive Timestamp:    0.000000000

Traffic seems to be from many different sources and not concentrated to any one source IP.

All these facts combined suggest the traffic is coming from: Human Activity on a Mobile Device (Phone/Tablet/Laptop)

The most popular Android applications (including Snapchat) were installed on a test phone and none of them generated NTP activity. The base Android system was seen to send NTP requests to 2.android.pool.ntp.org

iPhones and iPads were tested, and the iOS system only sent NTP traffic to time-ios.apple.com

Snapchat was installed on multiple iPhones, and this caused the iphone to lookup the following hostnames in DNS and send NTP to each unique IP returned:

0.pool.ntp.org
0.uk.pool.ntp.org
0.us.pool.ntp.org
asia.pool.ntp.org
europe.pool.ntp.org
north-america.pool.ntp.org
south-america.pool.ntp.org
oceania.pool.ntp.org
africa.pool.ntp.org

This matches the list embedded in this iOS NTP library: https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-lib/NetworkClock.m#L121

The NTP flags on the majority of NTP traffic also match the ones set here: https://github.com/jbenet/ios-ntp/blob/master/ios-ntp-lib/NetAssociation.m#L254

Snapchat has posted a fix to iTunes (Dec 20) and the update is slowly rolling out - https://mailman.nanog.org/pipermail/nanog/2016-December/089610.html

Credit for this information comes from a lot of different sources. Thanks to everyone that helped out!

3 Likes
2

I made a github issue on the library with the hardcoded pool.ntp.org zone names.

As people pointed out on IRC, using the continent names doesn’t make much sense in virtually all cases. That we have them is really mostly a historical artifact. :-/ (Probably it’d make sense to deprecate them on the website).

2 Likes
3

A post was split to a new topic: Software and devices without a vendor zone

4

Snapchat has posted a fixed application to iTunes (pending approval). I’ve updated the first post with that info.

1 Like
5

Reports on IRC says that the update is out now and in initial testing by Leo Bodnar it appears to not do NTP queries anymore (or at least more selectively…).

6

This graph shows the NTP traffic matching the ios-ntp library fingerprint on my NTP server:

client-types-ios.png900Ă—600 8.91 KB

1 Like
7

NTP Traffic today is down roughly 25% compared to yesterday, which is a good sign.

8

Snapchat added NTP servers to the Australia and Brazil pools last night as well - https://mailman.nanog.org/pipermail/nanog/2016-December/089620.html

1 Like
9

Tom Yates wrote a nice article about this for LWN, though I think he got the affect on the pool wrong.

The big zones managed okay in terms of server loss, but in particular South America and Africa were barely sustaining already and got decimated pretty badly.

The server count graph for Australia also looks pretty sad (it already wasn’t very good).

2 Likes
10

From the viewpoint of a server in the US pool, things are largely back to normal although traffic is still slightly elevated than prior to the event.

The graphs below show received packets per second and bandwidth in/out for a server in the US pool with “Net speed” 50mbps.

2 Likes
11

Hello everyone! (sorry for sort-of necro-ing)
I stumbled upon this fascinating event via the wikipedia entry.
That entry still represented the state of Dec 20th, i.e. in the heat of battle, before fixes were applied.

I took the liberty of bringing the wiki-entry up to date. I hope I’ve characterised the aftermath correctly.
I’d like it if someone more familiar to have a look at my changes, and update in case I missed anything or struck the nuances wrong. I’d hate to misrepresent the awesome work this community is doing!

3 Likes