Traffic in Singapore zone

Server operators
1

I have a server in the Sg zone and I noticed there is an explosion of traffic in the last few weeks.

981×382 25 KB

The server is a cheap VPS, which can handle about 20kpps at most. The pool speed was set to 50Mb and yesterday I set it to 25Mb. Still too much.

The number of servers in the zone didn’t change significantly.

Any idea what is going on there? Are other Asian zones impacted too? I don’t see anything in few European and North American zones.

From the large daily variations I assume most of the traffic comes from mobile apps. Maybe a popular one had an update recently, like the Snapchat incident few years ago?

EDIT: From values in the NTP requests it does indeed look like it is the same ios-ntp library. When looking at individual IP addresses, there is no apparent polling interval and ports are random. GeoIP says about 99% of the excessive traffic is coming from China.

2

There are several reasons for this to happen, since the server population in the zone is quite small, even a single server operator switching the servers e.g. from 1000Mbit to 100Mbit would significantly increase the traffic for all other servers in the zone.

Simple example: 50 servers, 5 at 1000Mbit, the others at 50Mbit on average.
If one server operator switches only 5 servers from 1000Mbit to 100Mbit the zone would experience a 10-fold of traffic for all other servers, while the total number of servers in the zone remains constant.

Other reasons:
You may check and compare the traffic ratio China mainland vs Singapore before the increase and after the increase.

Did the ratio between both regions change?
If yes: Sort the traffic by provider / device type / client and try to find which one caused the increase in traffic.

If no: Just lower the service bandwidth to an acceptable level for you.

If curious: Setup two servers, one mainland China, one Singapore, same settings, compare the traffic on both.

PS:

The server is a cheap VPS, which can handle about 20kpps at most.

Cheap is not always bad, but 20kkps? That´s really low. I got a PentiumPro machine (from 1995) handling 50kpps easily.

If you are paying for the VPS and it´s not e.g. one of the Google/Amazon free VPS plans available, I would likely switch to a more “robust” service.

3

Thanks for the response.

When a zone has a small number of servers, the speed setting has a smaller impact, because there is a fixed number of addresses that needs to be filled (4 for the non-numbered zone and 16 for the {0,1,2,3} zones).

In my case I had to go down to 1 Mbit to actually see a significant change in the traffic. The sg zone has about 20 servers. In the distribution of the servers in DNS I see that I got down to the bottom half. That might indicate the offending application uses a numbered zone.

There is clearly something wrong when such a small country receives so much NTP traffic, which increased quickly in the last few months, and most of it comes from a different country.

I think it’s the same issue as with Snapchat, just local to the zone. Maybe the application was developed in Singapore and is widely used in China. If we could download the most popular apps there and grep their content for sg.pool.ntp.org, maybe it would show something.

My server is limited by outgoing traffic per month, not the CPU. No point in upgrading to a better plan (there is no plan with unlimited traffic). I’m not interested in supporting companies abusing the pool. The terms of use are pretty clear.

4

I spun up a pool server in Singapore and also saw a large amount of traffic from China.
This uses ip2asn:
Percent
of total
23.8949 AS4134 CN CHINANET-BACKBONE No.31,Jin-rong Street
18.0032 AS4837 CN CHINA169-BACKBONE CNCGROUP China169 Backbone
12.5394 AS9808 CN CMNET-GD Guangdong Mobile Communication Co.Ltd.
12.514 AS16509 US AMAZON-02 - Amazon.com, Inc.
3.15873 AS24444 CN CMNET-V4SHANDONG-AS-AP Shandong Mobile Communication Company Limited
2.48071 AS24547 CN CMNET-V4HEBEI-AS-AP Hebei Mobile Communication Company Limited
2.37893 AS0 None Not routed
2.10875 AS56040 CN CMNET-GUANGDONG-AP China Mobile communications corporation
1.86282 AS24445 CN CMNET-V4HENAN-AS-AP Henan Mobile Communications Co.,Ltd
1.59451 AS56041 CN CMNET-ZHEJIANG-AP China Mobile communications corporation
1.4964 AS56044 CN CMNET-AS-LIAONING China Mobile communications corporation
1.289 AS3758 SG SINGNET SingNet

5

I throttled down the pool speed of IPv4 traffic to my servers somewhere around 21 april to do some testing. Will increase it again today. Please let me know if that is visible on your statistics.

6

Thanks, that might explain the increase I saw on the 21th. My traffic today is lower than yesterday.

Over the weekend, I tried few different speeds and observed traffic vs distribution of the server in DNS. It seems the application generating the large daily variations is actually using the non-numbered zone (4 addresses provided at a time). There are three servers with high speed setting (two of them are the Cloudflare servers) and the rest seems to have a much lower speed. My server apparently had a speed between those two groups and was selected frequently for that 4th slot. Removing or adding a single high-speed server could change the traffic significantly for the other servers, exactly as @Kashra said.

In any case, if we could identify that source, we would tell them to use a larger zone and ideally also reduce the rate of requests.

1 Like
7

I increased my capacity in Tokyo over the past few weeks and was wondering if this was noticeable on your end.

8

It might be nice to expose this on the site somehow (the graph also needs some work…), but the system does track a daily metric of the aggregate “active net speed”. There was a relatively large drop in available capacity in Singapore early in the year around when you posted this, @mlichvar.

select name, date, count_active, netspeed_active
from zone_server_counts zsc
inner join zones z on (z.id=zsc.zone_id)
where z.name = 'sg'
and ip_version = 'v4'
and date > '2021-02-01';

name,date,count_active,netspeed_active
sg,2021-02-02,27,11437920
sg,2021-02-03,28,11897920
sg,2021-02-04,28,11937920
sg,2021-02-05,27,11887920
sg,2021-02-06,29,12897920
sg,2021-02-07,29,12922920
sg,2021-02-08,30,12987920
sg,2021-02-09,26,11887920
sg,2021-02-10,30,12947920
sg,2021-02-11,27,12895920
sg,2021-02-12,28,12887920
sg,2021-02-13,28,12887920
sg,2021-02-14,28,11947920
sg,2021-02-15,29,12937920
sg,2021-02-16,27,11937920
sg,2021-02-17,28,11897920
sg,2021-02-18,28,11937920
sg,2021-02-19,28,11897920
sg,2021-02-20,25,10797920
sg,2021-02-21,24,10087920
sg,2021-02-22,25,10187920
sg,2021-02-23,25,9987920
sg,2021-02-24,24,9987920
sg,2021-02-25,25,10087920
sg,2021-02-26,27,10047920
sg,2021-02-27,25,9847920
sg,2021-02-28,26,9962920
sg,2021-03-01,25,9862920
sg,2021-03-02,29,10522920
sg,2021-03-03,26,10362920
sg,2021-03-04,27,10462920
sg,2021-03-05,26,10462920
sg,2021-03-06,26,10462920
sg,2021-03-07,26,10462920
sg,2021-03-08,25,10412920
sg,2021-03-09,26,10462920
sg,2021-03-10,25,10362920
sg,2021-03-11,25,10364920
sg,2021-03-12,24,10314920
sg,2021-03-13,26,10464920
sg,2021-03-14,25,10274920
sg,2021-03-15,23,9814920
sg,2021-03-16,24,9824920
sg,2021-03-17,24,9824920
sg,2021-03-18,27,10024920
sg,2021-03-19,25,9874920
sg,2021-03-20,23,9814920
sg,2021-03-21,25,9874920
sg,2021-03-22,22,9684920
sg,2021-03-23,23,9874920
sg,2021-03-24,23,9784920
sg,2021-03-25,23,9874920
sg,2021-03-26,22,9774920
sg,2021-03-27,22,9724920
sg,2021-03-28,23,9734920
sg,2021-03-29,21,9674920
sg,2021-03-30,22,9724920
sg,2021-03-31,21,9681920
sg,2021-04-01,21,9681920
sg,2021-04-02,22,9691920
sg,2021-04-03,22,9691920
sg,2021-04-04,21,9739920
sg,2021-04-05,21,9690920
sg,2021-04-06,21,9681920
sg,2021-04-07,22,9731920
sg,2021-04-08,20,9688920
sg,2021-04-09,21,9681920
sg,2021-04-10,20,9671920
sg,2021-04-11,20,9671920
sg,2021-04-12,21,9721920
sg,2021-04-13,20,9181920
sg,2021-04-14,20,9181920
sg,2021-04-15,19,9171920
sg,2021-04-16,19,9171920
sg,2021-04-17,19,9171920
sg,2021-04-18,21,9231920
sg,2021-04-19,20,9181920
sg,2021-04-20,19,9171920
sg,2021-04-21,19,8172304
sg,2021-04-22,19,8147304
sg,2021-04-23,18,8122304
sg,2021-04-24,19,8132304
sg,2021-04-25,19,8132304
sg,2021-04-26,19,8132304
sg,2021-04-27,19,8132304
sg,2021-04-28,20,8124304
sg,2021-04-29,19,8123304
sg,2021-04-30,21,8125304
sg,2021-05-01,21,8124816
sg,2021-05-02,29,14725804
sg,2021-05-03,22,9124804
sg,2021-05-04,24,9334420
sg,2021-05-05,23,9224804
sg,2021-05-06,23,9134804
sg,2021-05-07,23,9134804
sg,2021-05-08,22,9124804
sg,2021-05-09,24,9184804
sg,2021-05-10,24,9137804
sg,2021-05-11,24,9135804
sg,2021-05-12,23,9125804
sg,2021-05-13,22,9124932
sg,2021-05-14,22,9125420
sg,2021-05-15,22,8725188
sg,2021-05-16,22,8635188
sg,2021-05-17,21,8625188
sg,2021-05-18,21,8625188
sg,2021-05-19,21,8634188
sg,2021-05-20,22,8635188
sg,2021-05-21,22,8635188
sg,2021-05-22,22,8635188
sg,2021-05-23,22,8635188
sg,2021-05-24,22,8635920
sg,2021-05-25,22,8635920
sg,2021-05-26,23,8735920
sg,2021-05-27,22,8635920
sg,2021-05-28,23,9135920
sg,2021-05-29,23,9135920
sg,2021-05-30,23,9135920
sg,2021-05-31,24,9237420
sg,2021-06-01,22,9136420
sg,2021-06-02,23,9636420
sg,2021-06-03,24,9736420
sg,2021-06-04,24,9734420
sg,2021-06-05,22,9534420
sg,2021-06-06,23,9634420
sg,2021-06-07,23,9733420
sg,2021-06-08,23,9634420
sg,2021-06-09,23,9634420
sg,2021-06-10,23,9634420
sg,2021-06-11,25,9834420
sg,2021-06-12,23,9634420
sg,2021-06-13,24,9734420
sg,2021-06-14,21,9034420
sg,2021-06-15,22,9134420
sg,2021-06-16,22,9134420
sg,2021-06-17,24,9334420
sg,2021-06-18,23,9234420
sg,2021-06-19,24,9334420
sg,2021-06-20,22,9134420
sg,2021-06-21,22,9134420
sg,2021-06-22,24,9334420
sg,2021-06-23,24,9334420
sg,2021-06-24,22,9134420
sg,2021-06-25,22,9134420
sg,2021-06-26,24,9334420
sg,2021-06-27,21,9034420
sg,2021-06-28,22,9134420
sg,2021-06-29,22,9134420
sg,2021-06-30,24,9334420
sg,2021-07-01,22,9134420
sg,2021-07-02,22,9134420
sg,2021-07-03,22,9143420
9

The SG zone seems to be collapsing again. The number of active servers dropped significantly and I’m getting up to about 70 kpps in some hours of the day with the pool speed set to 1.5 Mbps.

4 Likes
10

I had to pull 2 / 3 due to https://community.ntppool.org/t/how-to-deal-with-invalid-abuse-reports/

The one remaining in @asia @sg is struggling along serving ~ 4 Mbps

I hope to bring something more substantial back to Singapore mid January.

Philippines zone is even worse. setting 512 Bkit results in 20Mbps of traffic as there is once again only Cloudflare in the zone.