Under-provisioned countries?

[gunter]

What countries currently exist that are under-provisioned in terms of bandwidth - not necessarily servers?

I have been experimenting past couple of weeks with many regions. Some regions with zero pool members do not necessarily even have the demand that could fill up 10mbit (Nigeria is one example).

On the other hand India and China of course could saturate easily 1gbit if joining the pool.

Recently discovered that Singapore - despite being a smaller country - can easily saturate 1gbit when joining the pool:

(this is on 75mbit netspeed)

Any other countries that are currently under-provisioned?

[n1zyy]

@gunter

I have been experimenting past couple of weeks with many regions. Some regions with zero pool members do not necessarily even have the demand that could fill up 10mbit (Nigeria is one example).

I suspect (but could easily be wrong) that what’s really happening is that queries for pool.ntp.org from Nigeria are returning servers from the larger region, rather than just the lone server in Nigeria, so it’s not seeing all the traffic.

I would put Malaysia on your list. I have a stratum 1 there and took it out of the pool because it’s at AWS and was costing me a fortune: pool.ntp.org: Statistics for 43.216.110.218

I brought up a VM at a more affordable place in Malaysia using it, and at the 1.5Mbps netspeed it’s been seeing about 5 Mbps. That said, there are 4 IPv4 servers in Malaysia and a query for the pool there always returns 4 IPs, so I may be seeing ¼ of the traffic. (Or maybe 100% if most clients are using the pool directive and using all 4.) I’m actually curious how the math will work out.

I’ve noticed that adding some basic rate limiting is dropping a lot of traffic. This is for my stratum 2 in Malaysia, but the trend fits other busy regions too:

chronyc> serverstats
NTP packets received       : 5552016284
NTP packets dropped        : 1562311238

So I’m surprised at how low the KoD responses are in your graph.

I no longer have a server in Brazil but I remember it was always surprisingly busy. I got kind of demoralized there because it was a spare EC2 instance and I realized that a huge volume of my traffic was from other EC2 instances and that I was effectively paying AWS to serve their customers.

I also have a Lightsail instance in Korea that is struggling to stay in the pool at the 1.5Mbps netspeed. Looks like it’s averaging about 1400qps. (I actually don’t entirely understand why it’s struggling at this speed, but that’s a separate issue.)

[gunter]

@n1zyy

I suspect (but could easily be wrong) that what’s really happening is that queries for pool.ntp.org from Nigeria are returning servers from the larger region, rather than just the lone server in Nigeria, so it’s not seeing all the traffic.

Yes, because configs by default will still require 3 other distinct servers. Still, the total demand would be just less than 50mbit for a whole country.

Currently my server is serving 100% of the whole region according to netspeed: pool.ntp.org: Statistics for 103.97.91.39 - but as predicted, only 1/3 is actually being routed.

I would put Malaysia on your list.

I appreciate your details and discovery. Interestingly I would have put Malaysia off my list initially as its rather smaller country. But I am guessing that its all traffic from India/China/Philippines? Not just Malaysia.

How did you get to run a Stratum 1 on AWS?

[n1zyy]

@gunter

Currently my server is serving 100% of the whole DNS according to netspeed: pool.ntp.org: Statistics for 103.97.91.39 - but as predicted, only 1/3 is actually being routed.

Oh! I have always been confused by that metric in the past. This is a really helpful illustration of it in practice.

As far as Malaysia, the VM that’s in the pool is mostly seeing Malaysia but neighboring queries do make up a bit of it: pool.ntp.org: Statistics for 111.90.143.178

@gunter

How did you get to run a Stratum 1 on AWS?

They make PTP available on new instance types in a few regions, and Malaysia jumped out to me as the one that doesn’t already have lots of public stratum 1 servers. I posted about it here a while ago: AWS Time Sync, now with (limited) PTP support

They don’t reveal a ton of information but from an earlier announcement about Time Sync, it seems like they have GNSS-locked PTP servers. I think there’s room for some academic debate about whether it’s really stratum 1 for a virtual machine that’s presumably a few steps removed from the GNSS receiver, but chronyd marks it as such and seems very happy with it:

chronyc> tracking
Reference ID    : 50484330 (PHC0)
Stratum         : 1
Ref time (UTC)  : Sun Jan 05 19:55:57 2025
System time     : 0.000000049 seconds slow of NTP time
Last offset     : -0.000000073 seconds
RMS offset      : 0.000000176 seconds
Frequency       : 14.522 ppm fast
Residual freq   : +0.000 ppm
Skew            : 0.223 ppm
Root delay      : 0.000010000 seconds
Root dispersion : 0.000001145 seconds
Update interval : 1.0 seconds
Leap status     : Normal

[gombadi]

@gunter
Hi. Yes Singapore is an interesting location for an NTP server. I found that I had to have the net speed turned right down or it would be flooded by requests. These were mainly from China while locations closer to China such as Tokyo and Seoul were getting far fewer requests from that country. May be a routing thing.
Recently I have been seeing a lot of requests from (I think mostly to - think DDoS) a Philippines ISP. I am seeing about 2.6 (up to 5.2) million requests per hour form/to a single ip address. I recently added rate limiting to Chrony to see what it would do. Would be interested to know if you are seeing similar things.
The Philippines is a location I would be interested in adding a server but have not, at this time, found a hosting company to use. Will continue to keep an eye on things.

@n1zyy
It is not only AWS that you can setup a stratum 1 server with. I recently set one up with Azure in Central India. It seems to be working fine as the pool monitoring is happy with it. The Azure setup seemed easier as it all just worked instead of having to compile/load kernel modules with AWS instances.
I see you have taken your Malaysia instance out of the pool because of costs. I have no plans to add my AWS(Tokyo)/Azure(India) instances to the pool even though India could do with more stratum 1 servers because the data transfer costs would be too much. I “may” allow other specific pool servers to use it as a source in the future but not wide open to the pool.
Another thing I saw while testing an AWS stratum 1 server in Malaysia was the routing. Not sure what was going on but the round trip time to Singapore was about 75msec (about 300km) while the round trip time to Tokyo was also about 75msec(about 5300km).

[n1zyy]

@gombadi

It is not only AWS that you can setup a stratum 1 server with. I recently set one up with Azure in Central India. It seems to be working fine as the pool monitoring is happy with it. The Azure setup seemed easier as it all just worked instead of having to compile/load kernel modules with AWS instances.

Whoa! I had no idea this was the case. I just spun an Ubuntu VM up, and right out of the box it’s doing PTP. That’s really impressive.

Both my Mumbai Lightsail instance and my Bangalore DigitalOcean droplet have selected it as the best. I tried both Pune and Chennai but both instances are preferring the Pune (“Central India”) one.

[gombadi]

I started out with a small burstable Azure instance but have recently changed to a non burstable instance to see if I see the same offset spikes. Only time will tell I guess.

[marco.davids]

@n1zyy

So I’m surprised at how low the KoD responses are in your graph.

You may like this simple tool I once wrote:

ntptools/ntptraf2 at main · mdavids/ntptools

[jnd]

From my limited experience it seems like most of Asia is in high demand of NTP service due to high population and perhaps higher popularity of Linux based systems. The problem is expensive traffic in the region. I contributed one server in India with 2 TB/month which is now doing fine set to 25 Mbit. Recently I added second one in Vietnam with 1 TB/month allowed traffic and right away I’m getting around 18 Mbit/s (9 up and 9 down) during the day on the lowest settings of 512 kbit. Which unfortunately means I consumed over third of the data for this month and will have to shut it down early at this rate. The sustainable average for my server would be 3 Mbit/s.

[Kets_One]

Collapse of Russia country zone

It is possible to set netspeed lower than 512k. Look at GET requests that ‘Manage servers’ page send. pool.ntp.org: the internet cluster of ntp servers will set 1 kbps. I set 1 kbps and it is easy to handle. upd: set 30 kbps, it gets higher - 50% cpu, about 70k ppm, ~70 mbps. Set 15 kbps, ~45 mbit/s, comfortable load.

See post above which might be interesting for you…

[davehart]

@Kets_One

See post above which might be interesting for you…

To make it a bit more explicit for those not routinely used to digging into web tech, go to the manage page where you can set netspeed and view the HTML source (right click on the background and choose View Source in Chrome). Locate the section around the IP address for which you want to lower netspeed, and grab the “auth_token” hidden input value, a long sequence of hexadecimal numbers with a single hypen at position 2. Copy it to the clipboard and paste it into a scratch document to keep it handy. Then using curl or the address bar in the browser, hit an URL like the following, substituting your own auth token value and matching server IP address:

https://manage.ntppool.org/manage/server/update/netspeed?netspeed=128&auth_token=2-02b7909e180c36216abc5215f48b5ba00fbb3989&server=IP_ADDRESS

The response should look like this if you’re successful:

{“netspeed”:“128 Kbit”,“zones”:“\n\n\nnorth-america\n\nus\n\n\n\n\n\n\n”}

Congratulations, you’ve just set the netspeed to 1/4 of the minimum the dropdown on the webpage allows.

[Sebhoster]

@gombadi

Recently I have been seeing a lot of requests from (I think mostly to - think DDoS) a Philippines ISP. I am seeing about 2.6 (up to 5.2) million requests per hour form/to a single ip address. I recently added rate limiting to Chrony to see what it would do. Would be interested to know if you are seeing similar things.

I had a virtual server in the Phillipines with Lightnode that was also constantly under load with upwards of thousands of requests per second from some IPs. The requests had strange patterns in them and came mostly from residential IP address blocks. I think those were probably caused by a broken NTP implementation on some device, but I was not motivated enough to investigate further. The topic was discussed here: Certain servers are not replying - #20 by Sebhoster

[mlichvar]

@davehart

The response should look like this if you’re successful:

{“netspeed”:“128 Kbit”,“zones”:“\n\n\nnorth-america\n\nus\n\n\n\n\n\n\n”}

Congratulations, you’ve just set the netspeed to 1/4 of the minimum the dropdown on the webpage allows.

Does this work also for increasing the speed beyond 1 Gb/s? I suspect this would be an easy way how attackers could take control of a zone.

Edit: I just tried it and it’s limited to 3 Gb/s, same as what can be selected in browser.

[n1zyy]

On the topic of under-served countries… I don’t quite know what happened, but the Malaysia pool appears to have collapsed in the past couple days. I was one of 4 servers for a while, but then I apparently became the only one, and then my provider suspended my account for “DDoS”. There is now one server back in the pool, which is not mine.

I don’t entirely understand the >50 MB/s spike here, though it does explain why the VPS host wasn’t thrilled. (I almost wonder if it was an actual attack of some sort.)