# Adding servers to the China zone **URL:** \ **Category:** Server operators\ **Created:** [January 4, 2017, 1:00am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88 "2017-01-04T01:00:05Z")\ **Posts on this page:** 20\ **Page:** 14
**Author:** ![allenZ](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/allenz/32/1258_2.png) [@allenZ](https://community.ntppool.org/u/allenZ)\ **Post date:** [March 4, 2019, 8:14am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/268 "2019-03-04T08:14:52Z")
Aha, I see my servers listed in @studentmain’s testing result , and with quite a loss rate. But that’s actually intended. I have implemented rate limit on my server, in `iptables` but not `ntp.conf`, to make sure my server could be removed from the pool periodically, so that I won’t run out of all my traffic and lost my other services on them at middle of a month (or to pay a thousand-dollar bill). I see @avij here is having similar problem and is using similar workaround as well. Actually, I’m quite agree with @LeoBodnar that servers of .cn pool should only been removed from the DNS response with total failures, but not high packet drop rate. Because the original problem here for .cn pool is that we have insufficient resource to handle all requests from the clients in this zone. Packet loss is just **not possible** to prevent. It’s clearly better to have all clients suffer a relatively low loss rate on requests to the pool, when there are multiple servers keep staying in pool to serve all the traffic (with some packets dropped), than servers have inbound traffic higher than the capacity get kicked out quickly, remains only very few ones serving all traffic, and every client suffers a high loss rate on requests. As I live in China myself, I know the pain from client side very well. I have st1 servers set up for own usage and for some of my servers inside China. But I just not able to provide those servers to pool, for I can’t afford to pay those bandwidth and traffic bills. This is an egg-chicken situation, the only way to solve it is to provide enough capacity of handling the entire zone, so that new servers can be added without been “DDoS” to death. We did so in OP, but it just didn’t worked out, because when we don’t really have enough resource in the pool, servers with packet loss (again, this is not possible to prevent in this circumstance) get kicked, then higher traffic leads to higher loss rate on other server, then more servers are kicked… then the whole zone collapsed. So how about we keep a constant amount of servers in the pool regardless of their dropping packets (only remove those don’t answer to any probes for long time, but keep ones that occasionally fail two or three probes), and gracefully add more servers to handle those requests, to finally reach the point that total capacity is larger than total requests, and servers in pool don’t need to run at full bandwidth, as other zones like .us or .eu are? ---
**Author:** ![felixonmars](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/felixonmars/32/274_2.png) [@felixonmars](https://community.ntppool.org/u/felixonmars)\ **Post date:** [March 4, 2019, 11:46am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/269 "2019-03-04T11:46:08Z")
> [@avij](#): > > Are we allowed to drop requests (by directing it to a server which can’t handle the traffic, perhaps in proportion of their bandwidth setting) if we can’t find a suitable server for responding to those, or should we try harder to find a NTP server for such requests, even if it means we would need to query a server that is far away from the client? I’d lean towards the latter myself. I’m afraid there is no simple answer. Connectivity is a complex thing, and packet loss to a monitoring station doesn’t always mean a traffic issue. The latter would work if the other server operators are happy about the unusual flow of traffic. I already saw quite some complaints here about too much Chinese traffic hitting like sg, hk, or tw pool. It might work better if the traffic was offload to a “super spare” zone like europe, but I doubt the server admins there will like the idea… The former however, will also break when the server is truly overloaded. I once tried to add my tiny machine in China, and only managed to connect to it again some time after its drop out of pool. But I can understand if this becomes an opt-in option for a server admin. The hard part would be how to prove that the server is really working. Of course more hands are needed this way… I also want to add that, I am a bit unsure of whether the current net speed weight setting is working as intended. From the descriptive text I would expect 1/1000 traffic from a server set to 1Mbps than one with 1000Mbps setting. If this assumption was true, I could already have all my tiny boxes added right now. But from the data posted above, a 384Kbps node still serve 60Mbps while my machine at 1000Mbps got 140Mbps. The ratio doesn’t really make a big difference at all for all those smaller boxes. If the traffic are scheduled by DNS rotate (that’s how I understand it), should a box with 1Mbps only appear once per 1000 resolves compare to the one with 1000Mbps setting? I would like to expect this even if we have only 2 hosts left in the zone. Please do correct me if I am wrong! ---
**Author:** ![dclabra](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/dclabra/32/412_2.png) [@dclabra](https://community.ntppool.org/u/dclabra)\ **Post date:** [March 4, 2019, 1:26pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/270 "2019-03-04T13:26:32Z")
We had to temporarily disable China traffic from our server 195.148.70.12 With bandwidth we have no problems but high amount of udp packets overloads our firewall. At the moment we don’t have possibility to bypass firewall. If we can’t find any reasonable solution (on this week) to get packets trough without overloading firewall, I post for server removal from China zone. ---
**Author:** ![alica](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/alica/32/337_2.png) [@alica](https://community.ntppool.org/u/alica)\ **Post date:** [March 4, 2019, 2:44pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/271 "2019-03-04T14:44:16Z")
> [@felixonmars](#): > > If the traffic are scheduled by DNS rotate (that’s how I understand it), should a box with 1Mbps only appear once per 1000 resolves compare to the one with 1000Mbps setting? I would like to expect this even if we have only 2 hosts left in the zone. This only works when there are tens of servers available. The current dns rotation always presents 4 servers in a query, unless the total available server count fells below 4. If we have only 4 servers available at this point, say 384kbps \*1 and 1000Mbps \*3, they will all be presented not regarding their initial registered bandwidth. Then the 384kbps one will receive unexpected high (1953x) traffic and fail. Output 4 servers in a row is never a problem in healthy zones, but for zones with single-digit servers available, this will effectively break the bandwidth setting. ---
**Author:** ![felixonmars](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/felixonmars/32/274_2.png) [@felixonmars](https://community.ntppool.org/u/felixonmars)\ **Post date:** [March 4, 2019, 3:16pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/272 "2019-03-04T15:16:27Z")
Yes, and I am proposing to change it to prefer the bandwidth preference over presenting 4 servers. I think this will resolve issues of other broken zones as well. ---
**Author:** ![Hedberg](https://avatars.discourse-cdn.com/v4/letter/h/e95f7d/32.png) [@Hedberg](https://community.ntppool.org/u/Hedberg)\ **Post date:** [March 4, 2019, 4:31pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/273 "2019-03-04T16:31:42Z")
One issue there is rarely mentioned is the spike in traffic everytime your IP is rotated into the DNS for even a short time. It is usually 20-30Mbps and I have observed much higher values. No normal equipment behind a NAT device etc can handle it and despite of the many NTP servers added yesterday there is only 3 active servers right now. ---
**Author:** ![avij](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/avij/32/197_2.png) [@avij](https://community.ntppool.org/u/avij)\ **Post date:** [March 4, 2019, 4:57pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/274 "2019-03-04T16:57:21Z")
If we don’t care about dropped packets, this would be “easy” to solve. Many ISPs don’t charge for incoming bandwidth, so just set up a few $5 virtual machines that swallow NTP packets at gigabit speeds but don’t send anything back. This would also need a cronjob at the pool monitoring server to “update servers set score=20 where class=‘notmonitored’;” or something to that effect. With a few of these in the pool the traffic for the other servers should be tolerable. BUT I don’t think doing that would be smart. I would rather try to use the capacity that we have elsewhere in the global zone. ---
**Author:** ![ddrown](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ddrown/32/11_2.png) [@ddrown](https://community.ntppool.org/u/ddrown)\ **Post date:** [March 4, 2019, 5:09pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/275 "2019-03-04T17:09:49Z")
To summarize what I believe are the challenges: 1. There is a significant amount of traffic to the cn zone - At least 150Mbit, probably more than 500Mbit based on reports in this thread 2. There is significant packet loss international -\> China - This affects traffic from the monitoring server to NTP servers in China 3. There is significant packet loss China -\> international - This affects client’s ability to reach international servers 4. There is currently far more client traffic than NTP server capacity 5. Typical VMs bought by individuals have 1TB/mo~20TB/mo bandwidth included - This is around 3Mbit~60Mbit - To handle this load with individual’s VMs, it would take around 170 of them 6. Handling more than 10kpps (~7Mbit) of NTP requires special software or hardware - For instance: [Getting beyond 10k qps?](https://community.ntppool.org/t/getting-beyond-10k-qps/757/28) - Firewalls also need to be considered carefully at speeds higher than this So, with all those challenges, what can we do? 1. Drop traffic when client load is greater than NTP server capacity - This could be done as an empty DNS response - This has its own problems 2. Have a much lower score threshold for the cn zone - This would allow services to stay in longer in the face of packet loss and add some additional capacity - This would also make it harder for servers to signal they wanted to leave due to billing or other reasons 3. Recruit a mix of high bandwidth and low bandwidth servers to handle the load - That’s how this thread started - The low bandwidth users would want some sort of way to limit their max bandwidth due to billing - Servers in China have a hard time staying in the pool due to issue #2 4. Setup a monitoring server in China - Who could provide a place for this to run? What I would like to see is a combination of all four. This would require work and co-ordination to complete, and that is a problem on its own. ---
**Author:** ![Hedberg](https://avatars.discourse-cdn.com/v4/letter/h/e95f7d/32.png) [@Hedberg](https://community.ntppool.org/u/Hedberg)\ **Post date:** [March 4, 2019, 5:23pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/276 "2019-03-04T17:23:43Z")
Short term: I think a good start would be to adjust the scores and penalty system. So you are not punished so hard for a missing packet now and than and long term look at a local monitoring server. The latter was also the plan last year. ---
**Author:** ![allenZ](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/allenz/32/1258_2.png) [@allenZ](https://community.ntppool.org/u/allenZ)\ **Post date:** [March 5, 2019, 3:57am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/277 "2019-03-05T03:57:15Z")
> [@ddrown](#): > > Setup a monitoring server in China > > - Who could provide a place for this to run? In a local LUG discuss group, there is someone willing to provide a server with direct access to st1 time source for monitoring station. But it’s unclear what the pool admins require a monitoring station to equipped, that they may not fully fit (or willing to fit) the requirement. I believe @felixonmars knows them better than me 🙂 ---
**Author:** ![avij](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/avij/32/197_2.png) [@avij](https://community.ntppool.org/u/avij)\ **Post date:** [March 6, 2019, 12:07am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/278 "2019-03-06T00:07:48Z")
> [@allenZ](#): > > I have implemented rate limit on my server, in `iptables` but not `ntp.conf` , to make sure my server could be removed from the pool periodically, so that I won’t run out of all my traffic and lost my other services on them at middle of a month (or to pay a thousand-dollar bill). I see @avij here is having similar problem and is using similar workaround as well. I believe our approaches differ in that when I activate throttling, only queries from the monitoring server get dropped. This means I’m still answering queries from other clients even when the throttling is activated, but because my score drops below 10, I’ll get fewer requests. ---
**Author:** ![Bryce](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/bryce/32/281_2.png) [@Bryce](https://community.ntppool.org/u/Bryce)\ **Post date:** [March 6, 2019, 1:35am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/279 "2019-03-06T01:35:44Z")
I’ve been out for a while so it will take me some time to catch up but I wanted to say thanks to whoever added a few of my servers to the CN pool ---
**Author:** ![Zygo](https://avatars.discourse-cdn.com/v4/letter/z/34f0e0/32.png) [@Zygo](https://community.ntppool.org/u/Zygo)\ **Post date:** [March 6, 2019, 5:07am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/280 "2019-03-06T05:07:25Z")
> To handle this load with individual’s VMs, it would take around 170 of them > The current dns rotation always presents 4 servers in a query, unless the total available server count fells below 4 Put these together, and there’s a possible solution: raise the DNS rotation minimum for the CN zone to 170. If there are fewer than 170 servers in the CN pool, and a query comes in from CN, combine the CN pool with some pool that has excess capacity (\*), so the CN nodes are 100% loaded instead of 5000% or 10000% loaded relative to their chosen netspeed. If people add a few servers capable of more than 10kpps that’s great–it brings down the load for the smaller ones–but it might be better to keep the raw IP count high to keep the zone robust against major outages. The minimum number of DNS entries should be different for each zone. 4 is probably fine for most zones in the world because most countries are relatively small, but the big ones like CN (or IN or US) will need much higher minimums. Even better would be to estimate both capacity (from netspeed settings) and request load (from DNS query data), and use that as the threshold rather than a raw IP count; however, a simple hardcoded “nodes\_in\_zone \> 170” threshold might help bootstrap CN in the short term. This might even be a permanent solution–we don’t expect the load from CN to go _down_ in the foreseeable future, so why would we remove that code? If the CN zone gets big and robust enough, we won’t need to raise the hardcoded limit either, the pool would naturally never be too small again. In theory, US needs a minimum number of servers much greater than 4 too, but US has always had hundreds, so this problem never comes up there. China is a big place. It should be easy to get 170 volunteers to run small VM servers there, but before that can happen, the NTP pool DNS service has to get really, really good at never, _ever_ hitting them with a super-sized DDoS cannon. I put my NTP servers in the US and CA pools because I was already running them for my own clients, so it cost me nothing to add them to the NTP pool and let other people use them too. If I ever got a bandwidth bill or a service outage from my participation in the NTP pool, I’d seriously reconsider my contribution, and probably drop at least some nodes out of the pool entirely. This kind of casual, zero-cost, opportunistic contribution that happens in other parts of the world should be possible in China too, and fixing the pool’s DNS server behavior when demand exceeds supply is a big part of making that happen. (\*) where’s the excess capacity? It’s tempting to say “hurl the packets at Europe/the rest of Asia/the world”, but if inter-zone packets are just going to get dropped, then it’s better to just blackhole them rather than impose costs on already stressed international network operators. In that case, the “larger pool” is just enough blackhole IPs to make the CN pool big enough that netspeed matches reality. Are there anycast blackhole IPs that could be used? 127.0.0.1? ---
**Author:** ![dclabra](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/dclabra/32/412_2.png) [@dclabra](https://community.ntppool.org/u/dclabra)\ **Post date:** [March 6, 2019, 9:29am UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/281 "2019-03-06T09:29:38Z")
You can remove our server 195.148.70.12 from china pool. ---
**Author:** ![Bryce](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/bryce/32/281_2.png) [@Bryce](https://community.ntppool.org/u/Bryce)\ **Post date:** [March 6, 2019, 3:37pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/282 "2019-03-06T15:37:37Z")
By the way, does anyone have any tips for optimizing the number of requests that you are able to handle? My server is running Chrony on Ubuntu because I could not get ntpd to work but I could try to switch if it would be faster. ---
**Author:** ![apuls](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/apuls/32/1467_2.png) [@apuls](https://community.ntppool.org/u/apuls)\ **Post date:** [March 6, 2019, 5:46pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/283 "2019-03-06T17:46:13Z")
@iocc please add to CN (remove from FR) [https://www.ntppool.org/scores/51.15.238.170](https://www.ntppool.org/scores/51.15.238.170) [https://www.ntppool.org/scores/51.15.241.78](https://www.ntppool.org/scores/51.15.241.78) [https://www.ntppool.org/scores/2001:bc8:4400:2c00::20:91f](https://www.ntppool.org/scores/2001:bc8:4400:2c00::20:91f) [https://www.ntppool.org/scores/2001:bc8:4400:2c00::34:b01](https://www.ntppool.org/scores/2001:bc8:4400:2c00::34:b01) ---
**Author:** ![iocc](https://avatars.discourse-cdn.com/v4/letter/i/ba9def/32.png) [@iocc](https://community.ntppool.org/u/iocc)\ **Post date:** [March 6, 2019, 8:02pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/284 "2019-03-06T20:02:47Z")
Some of them have score -100. I added two IPv4 servers. Thanks. The problem with the .CN pool isnt much of a bandwith problem, its packets per second and usually the hardware cant keep up with the packet rate. The peak to my servers has been 300 kpps/s. Its very underserved, we need to get back to around 50 servers for the .CN pool for it to be stable. ---
**Author:** ![iocc](https://avatars.discourse-cdn.com/v4/letter/i/ba9def/32.png) [@iocc](https://community.ntppool.org/u/iocc)\ **Post date:** [March 6, 2019, 8:11pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/285 "2019-03-06T20:11:35Z")
> [@LeoBodnar](#): > > Just 3 to 4 servers left in .cn > One of ours is working at 100% wire speed. > What was the plan B again? For plan B I would suggest to change the monitoring score for the .CN pool so that you would need ten “bad scores” instead of one too loose a point in the monitoring. Then a lot less servers would be kicked out and soon we hopefully have around 50 servers that is bare minimum for the .CN pool. I guess that would be a quite easy fix that can be implemented right now and doesnt cost anyone money. What do you think about that @ask? ---
**Author:** ![faelix](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/faelix/32/168_2.png) [@faelix](https://community.ntppool.org/u/faelix)\ **Post date:** [March 6, 2019, 8:53pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/287 "2019-03-06T20:53:49Z")
> [@LeoBodnar](#): > > I am pulling out from .cn on Monday in protest of stupid robot monitoring system whose only purpose now seems to be making pretty chart selfies of broken self. > I have been waiting for years but enough is enough. > > When monitoring is fixed I might be back. We did the same a couple of weeks ago, removing two NTP ECMP clusters each with several gigabit/sec of uplink (easily able to serve several hundred thousand requests/sec). It’s pointless hauling packets around the world from China when the problem is a mix of: - overloaded non-China nodes being kicked out of the CN pool (thus cascading the load onto other nodes which get overloaded, etc) - capacity between China and “mainland Internet” (so that nodes in China don’t get monitored correctly, and get withdrawn from the CN pool even though they’re probably serving clients in China just fine) I’ve tried to discuss this and raise this several times, suggested ideas, for what the NTP pool project might do to try to fix this. Nobody’s given any consensus, so I’m not about to start submitting PRs to the git repos to implement any of my proposals. ---
**Author:** ![iocc](https://avatars.discourse-cdn.com/v4/letter/i/ba9def/32.png) [@iocc](https://community.ntppool.org/u/iocc)\ **Post date:** [March 6, 2019, 9:32pm UTC](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88/288 "2019-03-06T21:32:17Z")
> [@David](#): > > Please add our server to CN pool > [https://www.pool.ntp.org/scores/185.255.55.20](https://www.pool.ntp.org/scores/185.255.55.20) Added, thanks! > [@NoahMcNallie](#): > > Please move [tick.hk.china.logiplex.net](http://tick.hk.china.logiplex.net) out of the Asia zone and into the cn zone and keep it in the hk zone. Ok, fixed! Thanks. > [@felixonmars](#): > > My server is currently serving 200kpps at ~180Mbps. I just added two more unmetered gigabyte servers with the same config to the pool, please add them to the cn zone too: > > [https://www.ntppool.org/scores/62.210.205.24](https://www.ntppool.org/scores/62.210.205.24) > [https://www.ntppool.org/scores/78.46.102.180](https://www.ntppool.org/scores/78.46.102.180) Thanks! Added, Im sure they will help. > [@Medowar](#): > > one more server for the china zone: > [https://www.ntppool.org/scores/116.203.67.76](https://www.ntppool.org/scores/116.203.67.76) > [https://www.ntppool.org/scores/2a01:4f8:1c1c:8038::1](https://www.ntppool.org/scores/2a01:4f8:1c1c:8038::1) Thanks. Added. > [@dclabra](#): > > You can remove our server 195.148.70.12 from china pool. Ok, removed. > [@apuls](#): > > please add to CN (remove from FR) > > [https://www.ntppool.org/scores/51.15.238.170](https://www.ntppool.org/scores/51.15.238.170) > [https://www.ntppool.org/scores/51.15.241.78](https://www.ntppool.org/scores/51.15.241.78) > [https://www.ntppool.org/scores/2001:bc8:4400:2c00::20:91f](https://www.ntppool.org/scores/2001:bc8:4400:2c00::20:91f) > [https://www.ntppool.org/scores/2001:bc8:4400:2c00::34:b01](https://www.ntppool.org/scores/2001:bc8:4400:2c00::34:b01) Thanks! Added .CN and removed .FR. > [@kennethr](#): > > @iocc Please add this server: [https://www.ntppool.org/scores/95.216.74.37](https://www.ntppool.org/scores/95.216.74.37) Sure. Added. [Previous page](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88.md?page=13) [Next page](https://community.ntppool.org/t/adding-servers-to-the-china-zone/88.md?page=15)