Adding servers to the China zone


#45

Done, thank you.

The capacity available to the China pool has gone from practically nothing to pretty reasonable over the last week and a half. Hopefully we’re getting to where it’s possible for an operator in China to add a server and manage the load by configuring the net speed.

select date,count_active as active,
count_registered as registered,
netspeed_active as netspeed
from zone_server_counts
where zone_id = 60 and ip_version = 'v4'
and date > '2016-12-01';

+------------+--------+------------+----------+
| date       | active | registered | netspeed |
+------------+--------+------------+----------+
| 2016-12-02 | 1      | 12         | 10000    |
| 2016-12-03 | 0      | 13         | 0        |
| 2016-12-04 | 2      | 12         | 11000    |
| 2016-12-05 | 0      | 13         | 0        |
| 2016-12-06 | 0      | 12         | 0        |
| 2016-12-07 | 3      | 12         | 200384   |
| 2016-12-08 | 2      | 12         | 200000   |
| 2016-12-09 | 2      | 12         | 110000   |
| 2016-12-10 | 1      | 12         | 1000     |
| 2016-12-11 | 0      | 12         | 0        |
| 2016-12-12 | 0      | 12         | 0        |
| 2016-12-13 | 1      | 12         | 10000    |
| 2016-12-14 | 0      | 12         | 0        |
| 2016-12-15 | 1      | 12         | 384      |
| 2016-12-16 | 1      | 12         | 10000    |
| 2016-12-17 | 0      | 12         | 0        |
| 2016-12-18 | 1      | 12         | 10000    |
| 2016-12-19 | 0      | 13         | 0        |
| 2016-12-20 | 1      | 14         | 100000   |
| 2016-12-21 | 1      | 14         | 100000   |
| 2016-12-22 | 1      | 14         | 1000     |
| 2016-12-23 | 4      | 14         | 300384   |
| 2016-12-24 | 2      | 14         | 100384   |
| 2016-12-25 | 2      | 14         | 110000   |
| 2016-12-26 | 1      | 14         | 100000   |
| 2016-12-27 | 2      | 14         | 110000   |
| 2016-12-28 | 3      | 14         | 120000   |
| 2016-12-29 | 2      | 14         | 100384   |
| 2016-12-30 | 3      | 14         | 120000   |
| 2016-12-31 | 2      | 14         | 101000   |
| 2017-01-01 | 5      | 14         | 210896   |
| 2017-01-02 | 2      | 15         | 1100000  |
| 2017-01-03 | 2      | 16         | 150000   |
| 2017-01-04 | 6      | 17         | 1160896  |
| 2017-01-05 | 32     | 42         | 12331384 |
| 2017-01-06 | 29     | 42         | 9743896  |
| 2017-01-07 | 33     | 43         | 10615396 |
| 2017-01-08 | 31     | 44         | 8385896  |
| 2017-01-09 | 33     | 44         | 10361896 |
| 2017-01-10 | 32     | 44         | 10351896 |
| 2017-01-11 | 28     | 44         | 9321408  |
| 2017-01-12 | 31     | 46         | 10132408 |
| 2017-01-13 | 34     | 47         | 9668408  |
| 2017-01-14 | 37     | 47         | 10765408 |
+------------+--------+------------+----------+

After adding the iocc servers the last numbers are:

2017-01-14	44	55	19745408

(So the existing servers should see a big decrease in the load and maybe be able to tweak their net speeds up a bit … :slight_smile:)


#46

U’re welcome.

I get 4500 - 6500 req/s to the server that are listed in cn.pool.ntp.org.
If not listed I get about 2500-4500 req/s. When only was in se and europe
zones I got about 500-2500 (low/high) req/s.

Thats 24000 - 40000 req/s avg in total of all 8 ntp-servers.


#47

You can add http://www.pool.ntp.org/scores/188.39.37.91 as well if you wish. :slight_smile:

EDIT: It appears the added traffic may have made some of our other hardware unhappy. Upgrading it shortly.
EDIT: HW replaced, HW upgraded. All is well. :smiley:


#48

Please add this one to the CN zone:
http://www.pool.ntp.org/scores/51.15.41.135


#49

Done!

How is the traffic looking for those of you with servers in the CN zone?


#50

5.103.128.88 recieves between 5-10Mbit/s


#51

Same for 176.9.118.9


#52

Averaging 4-5Mb/s for the past 3 weeks for 203.135.184.123.


suggests that India is a good next candidate to bootstrap. After that, going down the list of countries by population, Indonesia, Brazil, Nigeria, Mexico, Philippines, Vietnam all have low server counts. As before I’m happy to be in the vanguard of these efforts.


#53

suggests that India is a good next candidate to bootstrap. After that, going down the list of countries by population, Indonesia, Brazil, Nigeria, Mexico, Philippines, Vietnam all have low server counts. As before I’m happy to be in the vanguard of these efforts.

I discovered that the relation between net speed vs. packets per second is not fully linear with low amounts of servers in the pool. I upped from 100 to 1000 Mbit/s for Indonesia and packet rate increased from 2k per second to 3.7k per second which is still pretty easy to handle. At 100 Mbit/s I was probably already absorbing a large part of available NTP requests to the pool.

For India I have no figures for higher net speeds yet. I ran into significant packet loss problems at my India server for higher net speeds and have currently disconnected it from the pool while looking for a more reliable server provider. Distributing the load over more servers for the countries you mentioned is of course the best solution.


#54

5-10 Mbit/s. Its seldom over 12 Mbit, but it does happend.


#55

Please add this server to the CN pool.

http://www.pool.ntp.org/scores/163.172.177.158


Help! Pool Server DDoS! Has my IP got Hardcoded?!
#56

You are welcome to add http://www.pool.ntp.org/scores/88.96.199.9 to any zone which needs bootstrapping


#57

This server can be added to the CN pool.
http://www.pool.ntp.org/scores/51.15.49.133


#58

If there is need of one more server added to the CN zone, you can add this server: http://www.pool.ntp.org/scores/51.174.131.248


#59

Feel free to add these to the China zone as well:

http://www.pool.ntp.org/scores/94.237.64.20
http://www.pool.ntp.org/scores/2a04:3543:1000:2310:d862:f5ff:fe4e:6e9a


#60

Please take 88.96.199.9 out of the CN zone again - it’s killing my internet!
Every time the score rises above 10, the Chinese time synch load goes through the roof and my router grinds to a halt, for ALL traffic! Then it drops down to negative figures and the internet becomes usable again!
Took me a while to work out what had happened - it started at the beginning of the weekend.
Did another bunch on CN servers vanish, leaving a high load to me, or what?
Dropped the speed to try to control it, but it’s just cycling more slowly now (and of course I’m less available to local clients with the reduced speed setting).
Once I have stability, I’ll ramp the speed back up - it was at 10Mbit, now on the minimum without actually deleting it!
You can look at the scores on http://www.pool.ntp.org/scores/88.96.199.9 to see what’s happened.


#61

Phil, usually the packet rate itself isn’t a problem, but the problem is with routers trying to track all the connections to the thousands of individual IP addresses. On one of my servers serving the US and CN zones, I received a million queries from 466125 different IP addresses within 1 minute 27 seconds. Trying to track hundreds of thousands of connections might not end well.

If you can, you could turn off connection tracking for NTP requests in your router’s firewall. For reference, I have this in my settings on a CentOS 6 host:

/sbin/iptables -t raw -A PREROUTING -p udp --dport 123 -j NOTRACK
/sbin/ip6tables -t raw -A PREROUTING -p udp --dport 123 -j NOTRACK
/sbin/iptables -t raw -A OUTPUT -p udp --sport 123 -j NOTRACK
/sbin/ip6tables -t raw -A OUTPUT -p udp --sport 123 -j NOTRACK

The NOTRACK target seems to be deprecated in newer versions, so on CentOS 7 hosts I’m using this instead:

/sbin/iptables -t raw -A PREROUTING -p udp --dport 123 -j CT --notrack
/sbin/ip6tables -t raw -A PREROUTING -p udp --dport 123 -j CT --notrack
/sbin/iptables -t raw -A OUTPUT -p udp --sport 123 -j CT --notrack
/sbin/ip6tables -t raw -A OUTPUT -p udp --sport 123 -j CT --notrack

This stops the firewall from tracking connections to/from the server’s UDP port 123. Perhaps there’s something similar in your router that you could use.


#62

Would that it were so simple!
Unfortunately, the only 2 methods I have to access the router are through a stripped to the basics CLI - which provides no access at all to the filesystem and only a limited set of commands, or through it’s web interface which is even more restrictive (although of course, easier to use). It’s possible it MAY be running some version of Linux, but if so, it’s stripped to the basics, which a customised shell giving a very limited set of commands.
All I can find out about it is from these commands:
sysinfo - which just gives uptime, 1, 5 and 15 minute load averages, and memory use;
meminfo - which just gives a bit more detail on memory use;
swversion - 2.32e, which is singularly unhelpful, except to confirm that it’s on the latest stable release!
and deviceinfo, which just gives the model name (BiPAC 8800NL), SW version (as above) and MAC address.
ifconfig gives a *nix type output, and ps gives a list of very *nix looking processes, so if I could access the filesystem and edit it’s contents, that may be of some use. I backed up the config to a local drive, but it appears to be binary, and therefore not amenable to editing!
Sadly, I can’t even find /sbin - there doesn’t appear to be any way to get under this protective shell (which I suppose makes sense for a device of it’s type and probably reduces support costs for Billion, who make it).
I’m not sure how long I was in the CN zone for before this started happening, but I see another thread where an operator was getting flooded from there, so I don’t appear to be the only one, and absent a router which gives more access to it’s operating system, there doesn’t appear to be any way I can participate in the attempts to bootstrap the ntp pool in China, and I’m effectively knocked out of the pool until I’m removed from the CN zone.
If it were my firewall (another box entirely - I have a /29 network, so there are things outside the firewall as well as inside it) it would be a different matter! That’s running FreeBSD with fw2, and I can access anything I want, if I use root to do so! But that is running very happily - it also happens to be my stratum 1 ntp server, with a Garmin 18x plugged into it’s serial port (for signalling) and USB port (for power).
I’ve never yet seen it get above 10% processor utilisation, and the only time I ever had a a problem was when a logfile got left out of the archival rotation, and filled up /var (oops).


#63

Phil, I think there might be some sort of a firewall running on your router as well, even if you’re not actively using it. Depending on how your network is set up, you might be able to simply disable the firewall functionality in the router without compromising the security of your network. This could probably be done from the router’s web user interface.

Obviously any “disable the firewall” advices should be taken with a grain of salt, but I believe you are familiar enough with networking concepts to be able to consider this option and its consequences.


#64

The only “firewalling” (which they refer to as IP filtering) on the router is the option to route packets to the internal network (i.e. my /29 subnet). If I switch it off, it doesn’t route anything at all (despite the routes being in the routing table) - and yes, I was also confused that this was only enabled within the IP filtering rules - so much so that it took a call to tech support to get the router to function as one, rather than a NAT gateway, bridge, or simply a doorstop! The only way to allow packets into and out of the subnet is to have a rule in place allowing it, otherwise all traffic is blocked, even with the firewall switched off (well, the filter table empty, which is as close as it will allow), despite routes being in place in the routing table! No other rules are in place, so no real firewalling is happening - all packets for my subnet are simply passed on to the relevant device behind the router, anything from inside that is bound for anywhere else gets thrown up the wire (and fibre, once it gets to the cabinet in the street) to my ISP to sort out! Of course, if it isn’t supposed to be coming to my subnet it shouldn’t be arriving at the router on the external (VDSL) interface at all, but that is for my ISP to manage!
Bizarre, I know, but that is how the software on it works (or doesn’t, if the Chinese are hammering it with NTP requests)!

It appears to me that some Chinese DNS servers cache DNS lookups to the pool, and only regard a pool record as stale if the site becomes unreachable, at which point they look for another victim. Of course, this caching breaks the load balancing of the pool!
What I’ve seen is that following my addition to the CN zone in an attempt to kickstart it, it took a short while before my IP happened to get handed to their DNS, at which point everyone using that DNS server in China started beating on my IP address asking for the correct time! This knocked my router (and whole subnet) offline until the pool saw that I’d gone, removed me as my score was below 10 (it used to be pegged at 20) and the Chinese gave up and asked for another IP address. Then my network recovered, and ran fine, until whatever random time after reaching a score of 10 my IP address happened to get handed to them again. Rinse and repeat. Going by another thread on here, I’m not the first to suffer from this, which tends to confirm that it is not a problem with my setup (which was working fine until I volunteered to join the CN zone to bootstrap it, I think we may be finding out why it needed bootstrapping in the first place!
Of course, it may only be a subset of the DNS servers in China which are behaving in this way, but it seems to be quite enough to cause a problem, and is probably the reason for so few NTP servers being in the CN zone, relative to the size and population of the country - this DNS caching keeps knocking them over, so it’s only with the aid of some rate limiting filter that any NTP server can survive in the CN zone.