Traffic in Singapore zone

I have a server in the Sg zone and I noticed there is an explosion of traffic in the last few weeks.

The server is a cheap VPS, which can handle about 20kpps at most. The pool speed was set to 50Mb and yesterday I set it to 25Mb. Still too much.

The number of servers in the zone didn’t change significantly.

Any idea what is going on there? Are other Asian zones impacted too? I don’t see anything in few European and North American zones.

From the large daily variations I assume most of the traffic comes from mobile apps. Maybe a popular one had an update recently, like the Snapchat incident few years ago?

EDIT: From values in the NTP requests it does indeed look like it is the same ios-ntp library. When looking at individual IP addresses, there is no apparent polling interval and ports are random. GeoIP says about 99% of the excessive traffic is coming from China.

There are several reasons for this to happen, since the server population in the zone is quite small, even a single server operator switching the servers e.g. from 1000Mbit to 100Mbit would significantly increase the traffic for all other servers in the zone.

Simple example: 50 servers, 5 at 1000Mbit, the others at 50Mbit on average.
If one server operator switches only 5 servers from 1000Mbit to 100Mbit the zone would experience a 10-fold of traffic for all other servers, while the total number of servers in the zone remains constant.

Other reasons:
You may check and compare the traffic ratio China mainland vs Singapore before the increase and after the increase.

Did the ratio between both regions change?
If yes: Sort the traffic by provider / device type / client and try to find which one caused the increase in traffic.

If no: Just lower the service bandwidth to an acceptable level for you.

If curious: Setup two servers, one mainland China, one Singapore, same settings, compare the traffic on both.

PS:

The server is a cheap VPS, which can handle about 20kpps at most.

Cheap is not always bad, but 20kkps? That´s really low. I got a PentiumPro machine (from 1995) handling 50kpps easily.

If you are paying for the VPS and it´s not e.g. one of the Google/Amazon free VPS plans available, I would likely switch to a more “robust” service.

Thanks for the response.

When a zone has a small number of servers, the speed setting has a smaller impact, because there is a fixed number of addresses that needs to be filled (4 for the non-numbered zone and 16 for the {0,1,2,3} zones).

In my case I had to go down to 1 Mbit to actually see a significant change in the traffic. The sg zone has about 20 servers. In the distribution of the servers in DNS I see that I got down to the bottom half. That might indicate the offending application uses a numbered zone.

There is clearly something wrong when such a small country receives so much NTP traffic, which increased quickly in the last few months, and most of it comes from a different country.

I think it’s the same issue as with Snapchat, just local to the zone. Maybe the application was developed in Singapore and is widely used in China. If we could download the most popular apps there and grep their content for sg.pool.ntp.org, maybe it would show something.

My server is limited by outgoing traffic per month, not the CPU. No point in upgrading to a better plan (there is no plan with unlimited traffic). I’m not interested in supporting companies abusing the pool. The terms of use are pretty clear.

I spun up a pool server in Singapore and also saw a large amount of traffic from China.
This uses ip2asn:
Percent
of total
23.8949 AS4134 CN CHINANET-BACKBONE No.31,Jin-rong Street
18.0032 AS4837 CN CHINA169-BACKBONE CNCGROUP China169 Backbone
12.5394 AS9808 CN CMNET-GD Guangdong Mobile Communication Co.Ltd.
12.514 AS16509 US AMAZON-02 - Amazon.com, Inc.
3.15873 AS24444 CN CMNET-V4SHANDONG-AS-AP Shandong Mobile Communication Company Limited
2.48071 AS24547 CN CMNET-V4HEBEI-AS-AP Hebei Mobile Communication Company Limited
2.37893 AS0 None Not routed
2.10875 AS56040 CN CMNET-GUANGDONG-AP China Mobile communications corporation
1.86282 AS24445 CN CMNET-V4HENAN-AS-AP Henan Mobile Communications Co.,Ltd
1.59451 AS56041 CN CMNET-ZHEJIANG-AP China Mobile communications corporation
1.4964 AS56044 CN CMNET-AS-LIAONING China Mobile communications corporation
1.289 AS3758 SG SINGNET SingNet

I throttled down the pool speed of IPv4 traffic to my servers somewhere around 21 april to do some testing. Will increase it again today. Please let me know if that is visible on your statistics.

Thanks, that might explain the increase I saw on the 21th. My traffic today is lower than yesterday.

Over the weekend, I tried few different speeds and observed traffic vs distribution of the server in DNS. It seems the application generating the large daily variations is actually using the non-numbered zone (4 addresses provided at a time). There are three servers with high speed setting (two of them are the Cloudflare servers) and the rest seems to have a much lower speed. My server apparently had a speed between those two groups and was selected frequently for that 4th slot. Removing or adding a single high-speed server could change the traffic significantly for the other servers, exactly as @Kashra said.

In any case, if we could identify that source, we would tell them to use a larger zone and ideally also reduce the rate of requests.

1 Like