Steady traffic increase BE since around midnight CET yesterday

Server operators
1

On my server in the Netherlands that is helping out in the BE zone, I have seen a steady increase in traffic since about midnight local time yesterday, more than doubling so far, see attached throughput graph.

vnstat (53)
vnstat (53)500Ă—250 2.16 KB

Anyone else seeing this?

I have not had time yet to fully investigate, but unique IPv4 clients from BE currently outnumber clients from the NL zone by a factor of about 28, NTP IPv4 traffic is about 95% of all NTP traffic, and NTP traffic seems to by far outweigh any non-NTP traffic. So it currently looks to me as if NTP IPv4 traffic from the BE zone is what is causing this increase.

While it is not clear what is causing this, I have removed the server from the pool, see the dip at the end of the graph.

Seeing the steady increase, this doesn’t look like it’s a few misbehaving clients, also underpinned by looking at the traffic and not seeing any single client, or small number of clients, that would dominate.

At the same time, the traffic seems to be rather unresponsive to the server being removed from the pool, suggesting the majority is being generated by NTP clients, rather than SNTP clients. (I know that how traffic responds to a server’s changes in netspeed/removal from the pool can greatly vary between zones, but I don’t think I’ve ever seen such relatively small responsiveness to such action before.)

Putting it all together - the timing, the steady increase, the relative unresponsiveness to server removal from the pool - my current hypothesis is that some vendor/ISP was rolling out new firmware/software to devices, replacing an SNTP client in such firmware/software by an NTP client. Also inspired by the finding of how NTP traffic would increase with Ubuntu switching from systemd-timesyncd as default (S)NTP client to chronyd.

2

As your server serves both NL and BE, I’ll only add that I’m not seeing any extra traffic on my server that serves the NL zone only.

Edit: Unexpectedly, my server serving the Singapore zone shows elevated request rates for today. Nothing on my .fi or .cn servers.

2 Likes
3

I have a server that is only in the BE zone. The plot below shows NTP mode 3 request rate per minute. There is some diurnal variation, but I don’t see the 2025-12-13 growth you report.

image
image1176Ă—446 12.3 KB

1 Like
4

I told you it would happen. As @EU servers don’t step in to help.
This is a pool bug.

Zone servers don’t join. Never.

5

Bas, did you notice an increase in traffic on your .be server? That’s what is discussed here.

1 Like
6

Thanks @avij and @stevesommars for your feedback. Supports that it was specific to the BE zone, and potentially specific to my server. I have an IPv6-only server sitting in both zones as well, and it didn’t show anything, either. So not related to IPv6, either it seems.

In the meantime another interesting event: At some point, the amount of traffic dropped noticeably and very sharply.

vnstat (55)
vnstat (55)500Ă—250 1.73 KB

So my best current guess would be that some caching (or even authoritative) DNS server held on to one set of records for longer than the TTL indicates, and passed it out to clients, by and by directing more and more clients to my server over time. And obviously doing so even once my server had been removed from the pool.

One thing, though, is why apparently only my server was affected? As the pool typically hands out four records for a zone query, there should have been another three servers affected, but interestingly no report so far.

Anyway, I will keep my server out of the pool for the time being, and slowly ramp up the netspeed again once the traffic has reached a more “normal” and predictable level.

1 Like
7

I stopped all my servers to run 50Mbit, as I have asked several times to put ALL my servers to be included in the BE domain.

No reply, no changes. Looks like nobody cares.

Sorry, but I see no improvement in the pool to fix the ZONE-BUG…or allow people to support other countries themselves.

Why is this problem still here? I set them 6mbit or below, as It’s stupid they keep being attacked massively. You can see this in the numbers…zones are NOT used to help one bit.

@ask when are you going to fix this bug? I have asked this so many times, it’s not funny anymore.

8

While it is not clear what is causing this, I have removed the server from the pool, see the dip at the end of the graph.

I think that maybe, just maybe, they are changing GeoDNS to avoid this in-country geolocking.
(there’s no new version of GeoDNS ATM on github, so not sure). I checked the Belgium subzone size and it has not increased a lot lately, say, by adding servers from the NL.

For the technical point of view, this makes tons of sense – clients in BE can get good time services from DE NL and FR, and other neighbors. We looked into it in a research paper.

I did a quick experiment here from the Netherlands. I queried like 100 times for be.pool.ntp.org directly to their auth servers.
In other words, to know what servers are serving Belgium folks.

Results

You can see that DE, CH and NL are serving Belgium-based folks ( Geolocation for US servers is bogus given this service is Cloudflare, which is anycast, so geolocation does not work )

{'US': {'162.159.200.1', '162.159.200.123'},
 'BE': {'109.68.160.220',
  '185.111.204.220',
  '185.153.43.4',
  '185.21.134.11',
  '185.77.12.9',
  '185.89.20.5',
  '193.104.37.238',
  '193.121.15.225',
  '44.31.68.23',
  '45.87.76.3',
  '45.87.77.15',
  '45.87.78.35',
  '80.200.247.170',
  '91.181.42.222'},
 'DE': {'178.215.228.24'},
 'CH': {'156.106.214.48', '156.106.214.52'},
 'NL': {'185.92.222.26'}}

This is good. This is pretty much what Apple, Microsoft, and AWS do.

So technically, it’s good for the Pool.

Policy implications

While it is not clear what is causing this, I have removed the server from the pool, see the dip at the end of the graph.

I think folks have become used to server their own country-fellow folks.. I think you are now serving your EU neighbors, which I think it is also fine. But that is a personal choice..

Other alternatives

current hypothesis is that some vendor/ISP was rolling out new firmware/software to devices, replacing an SNTP client in such firmware/software by an NTP client.

I cannot prove or disprove that, but I think this is quite a difficult change in such short notice. The DNS experiments above suggest these are changes on " modernizing" the pool and stop with this in-country locking.

2 Likes
9

Thanks, @giovane, for your feedback! A few thoughts.

In my understanding, the GeoDNS implementation is already quite flexible to cover different scenarios, based on how it is parametrized/configured. The primary challenge to me seems to be how the zones it is given to serve have been populated in the first place, which is what the NTP pool code is responsible for. And in my understanding, that is currently still being done based on the “traditional” zone concept (country and continent zones, and unfortunately the biased handling of the two IP versions).

Whether the GeoDNS implementation will eventually need some updates/tweaks needs to be seen. But the primary challenge is to actually get anything done at all in this area. It isn’t that Ask didn’t himself see the need for changes already. In fact, at some point, when there was still a bit more active discussion on the topic a while back, he seemed to already have some ideas how to do it/what to do, and it was kind of on his agenda already (though some small other items to take care of first…).

Unfortunately, it seems other items always got higher priority so far, so given the little time he has to do actual development work on the project, this has been slipping for a while now, and like for the IPv6 topic, he doesn’t even engage on that topic anymore, or share any plans.

So as much as I would like to see changes in this area, whatever may be required to effect them (GeoDNS itself, or how to populate the zones, or…), I am not really sure there is anything going on at the moment yet.

It would be rather welcome if that were an indication of forthcoming changes in the pool system. However, I fear, this is just the outcome of some people asking to have the BE zone added to their servers manually to help out there, as is documented in other threads in this forum. I think besides myself, there’s at least two or three other people out there who had the BE zone successfully added to their servers, precisely from the zones that you highlight.

Sure. But just like with the IPv6 topic, when it was still being discussed more actively, when there was still some prospect that something would change in the not too distant future, there wasn’t a shortage of proposals how to do this properly, including how to properly manage the transition so that, e.g., people who only want to serve their own region, or who only can serve a certain amount of traffic, do not suddenly get requests from all over the place, or suddenly get flooded by requests when their own zone originally was rather quiet before.

And after all, weren’t you yourself advocating for changes in that area, if maybe for primarily other reasons (diversity and resilience for clients)?

Not long ago, a change in one product brought down the entire RU country zone.

I wish that were the case, but see above, I don’t really see that happening, and the effect you draw hope from having some other background…

1 Like
10

I can’t prove this afterwards as I don’t have any logs, but one possible scenario which might explain this event is if one of the nine authoritative DNS servers for pool.ntp.org got stuck serving the same data for Belgium. First drop in the traffic = 8 DNS servers stopped serving your IP. Second drop in traffic = the one stuck DNS server stopped serving your IP. If this happened, there should be NTP servers serving BE that a) received more traffic (like yours) and b) received less traffic. The B situation might be more difficult to notice compared to A.

1 Like
11

The primary challenge to me seems to be how the zones it is given to serve have been populated in the first place, which is what the NTP pool code is responsible for.

True, you can also add servers to other coutnry zones.

Take pool.ntp.org: Statistics for 178.215.228.24, which servers Belgium as per above:

  • it serves these countries; Zones: @ be de es europe nl pl uk âś“

And in my understanding, that is currently still being done based on the “traditional” zone concept (country and continent zones, and unfortunately the biased handling of the two IP versions).

Maybe they are slowing changing that, making some servers to serve multiple countries.

So I also don’t rembemer this being discussed here, but it was something they needed to fix anyway.

12

As your server serves both NL and BE, I’ll only add that I’m not seeing any extra traffic on my server that serves the NL zone only.

You can actually see on the server scores page which clients (well, DNS resolvers) is your NTP server assigned with.

Client distribution
Global points 3.80‱

Top Countries
sg	302.88 ‱	52.14 ‱ (5.81x)
cn	5.54 ‱
id	6.64 ‱
br	1.77 ‱
us	0.37 ‱

Your server is assigned to these zones: Zones: @ asia sg âś“
, whereas @ is the global zone.

I have no idea why then you are serving these other clients. There must either be another geodns version running or the zones here are not really reflecting what is going on

13

If you are wondering why my server that primarily serves the SG zone also has clients from CN, ID, BR and US listed, that’s because the server is also in the global @ zone. My server that serves primarily the NL zone (and the global @ zone) lists DE, US, GB and ES clients.

I don’t think there’s anything mysterious in that regard. It’s working as intended. Or I’m misunderstanding something in your message.

14

There’s been an extensive discussion on this in this forum not too long ago, but a few things indeed remain a mystery at this time.

1 Like
15

That is not what I was talking about. You said that the GeoDNS part of the system needs to change to allow more flexible handling of zones. My point only was that the GeoDNS service only serves what it is given, matching clients to statically filled zones based on several different potential “geo”-criteria, of which I believe only a subset is currently used today (country, continent). (And it handles the netspeed part.) The mapping of servers to zones is handled in another part of the system, which also takes care of statically adding servers that are registered in multiple zones to each of those.

Oh, I wished you were right, but I am not as optimistic as you are. Particularly because of all the challenges the change would entail, not least the one you raised. So I don’t think this could be introduced quietly without there being some information towards the community.

The topic comes up time and time again in threads that are often actually about the symptoms of the issues that the current strict zone assignment brings with it, or other related contexts (e.g., discussions why one sees clients from outside one’s own zone), so it’s rather spread, one needs to follow various threads to get everything that was discussed on the topic. The last I remember/could find quickly where Ask discussed some of his thinking was a thread you were involved in as well, around the time of you publishing your paper on GeoDNS mappings. At that time, since there seemed some interest from Ask’s side, there still was a lively discussion. That has mostly died down since then, only flaring up here and there, with more and more (or often very similar) ideas being raised as to how to dissolve the current strict country-zone-based setup to something more fluid, and how to manage the transition.

EDIT: Actually, I just realize that Ask sharing his thoughts on the topic was actually almost a year before you published your paper. Well, time flies…

16

Excellent, indeed, that looks like a reasonable explanation, thanks for spelling it out!

Indeed I hadn’t though about that some servers would have had to see less/a decrease in traffic, but that is obviously not as noticeable as an increase with its potential for trouble, as you mention.

While I also mentioned that myself, I am not so sure anymore though whether it would be really one of the authoritative name servers for the Pool that would have gotten “stuck” in this case. That is because while there is a limited number of names for those name servers (the nine you were referring to), each one points to a set of in my understanding independent actual server instances (at least there’s at least 30 individual IP addresses per IP protocol version that one gets from the names). I.e., not only a single authoritative server would have needed to get stuck, but several of them at the same time. Not impossible, especially while we don’t know what could have caused it. But perhaps less likely than a single DNS server at some ISP somewhere getting stuck.

17

Actually, I was wrong about that: Here is a rather recent statement by Ask on the topic.

Sounds a bit like the current country-based zoning system might stay, but the zones get populated more flexibly, backfilling from other zones as needed.

But let’s see…

2 Likes
18

Yeah… this starts to become one of many similar discussions on this topic we have had over the years. Too bad it mostly ends the same way.

The strange thing is ofcourse that to alleviate some of the most serious symptoms of the current GeoDNS allocation based on country/continent just more flexibility regarding country/zone assignments would help greatly in my opinion.
For example by letting the server owner select the zone he/she wants to serve or (if this is too scary for some reason) have pool management show more flexibility towards change requests of the same.

1 Like
19

, that’s because the server is also in the global @ zone.

All servers are part of the global zone. But feel countries are mapped to the global zone (South Sudan, Antarctica, see Fig 6c in this pdf

So in short: no ,being part of the global zone does not mean you’ll serve clients globally – at least back when we did the evaluation

20

) The mapping of servers to zones is handled in another part of the system, which also takes care of statically adding servers that are registered in multiple zones to each of those.

You’re right, you can do it two ways: either change geodns OR change the way you populate the zones.

But i forgot one important detail: users say in Belgium can bypass GeoDNS’es mapping and use nl.pool.ntp.org