Another NTP client failure story

Are they routing all these devices through their own network?

I posted some stats in another thread, I was getting around 500k/day from their netblock, US server 100Mb setting… Which equates to around 5-6 a second, which is about 1/10th of @mlichvar and that makes sense as he’s at 1Gb setting for the pool. Likewise for @mnordhoff that was at 23/s @ 500Mb (granted his sample time was just those 43 seconds, but it is still close enough to mlichvar’s to confirm overall rate).

If we are to extrapolate from a chart mlichvar posted in another thread, that a single 1Gb server handles 0.95% of the US traffic, and they are querying the whole pool equally (which appears to be the case)… So they are hitting just the US pool at: 50 / 0.95% * 86400 = 454,736,842 queries per day…

Likewise from the chart, the US pool receives about 389,474 queries per second, which puts just Jasper at using about 1.4% of that total usage (assuming they are only querying the US pool). If they are querying the global pool, which receives about 509617 QPS, then that number drops down to 1%… I estimated 1% from my own server stats, which I do receive traffic from all over the world, so that would fall in line with the above estimates…

One company, creating 1% of the total GLOBAL pool traffic… Does the pool serve only 100 clients? No… It serves hundreds of millions…

Last time I posted numbers my server (@ 100Mb) was seeing ~2.4 Million unique IPs a day… A 1Gb server would theoretically see about 24 Million (Maybe @mlichvar has stats he can post on daily unique IPs on his servers) … I’m not sure the 0.95% would hold true as that would be about 2.5 Billion IPs, which is over half the IPv4 address space. There would be a fair amount of overlap as the same devices would be querying multiple servers in a day (either regular polling via NTP, or some regular polling interval doing SNTP)… If you divide by 5 (as a rough guess of how many servers one IP might query in a day), you get around 500 Million IPs which is more believable from the existing numbers.

The only reason it’s not a DDoS is because the queries are coming from a relatively small group of IPs. But as I pointed out they are creating a very noticable amount traffic. Any NTP server which has the ‘limited’ statement in its default configuration (which is probably every one), is dropping packets from the Jasper 204.156.180.0/22 netblock. I believe the default ‘discard’ average query rate is 8-seconds (which is more than generous for how NTP should operate). Their query rate per-IP is way too high, plain and simple. Even if traffic continues to grow from those IPs, the only thing that is going to happen is more packets are going to get dropped. That’s assuming the pool servers are using the regular NTP distribution, I know nothing about Chrony, but I’m going to assume it’s very similarly configured / rate limited. Also, it’s not all the ~1,000 IPs in Jasper’s netblock, last time I logged it was around only 140 IPs creating all the traffic…

I just re-enabled logging this morning… I’ll let it run for a day and PM you some specific IPs and traffic numbers. That might help determine exactly what is going on.

*Note: Any mistakes or things that don’t make sense above is because I’m still drinking my coffee this morning… lol.

2 Likes