Adding servers to the China zone


#247

For those who might need this: I have two servers in the .cn pool, but their monthly bandwidth quota is 4TB for each. To make sure I don’t exceed the quota I need to throttle the traffic somehow. I do this by blocking probes from the monitoring server’s netblocks. I don’t block the traffic originating from elsewhere. The scripts I use are available at http://miuku.net/ntp/limit-speed.html

Hopefully this helps someone.


#248

Then you also agree that if everybody followed your advise and changed their settings to the same minimum, then you would receive more traffic since your share of the total resources would rise? Making it even more impossible for newcommers to help.


#249

Fresh data in China here, you can compare it with official monitoring system.

Raw data, about 213k records: csv mysql

Data collected in a server located in UESTC (Unfortuneately, we have no equipment to set an official monitoring station). Data start at 2019-03-02T05:57:53.634Z, end at 2019-03-03T04:48:35.146Z.


#250

No. If you had bothered to read my message to Leo, you would have noticed that the goal was to adjust his netspeed lower so that his servers would serve more traffic, due to his servers getting kicked out of the pool less frequently with the more appropriate netspeed setting. This would help reduce the traffic that would need to be sent to other pool servers.


#251

My server is currently serving 200kpps at ~180Mbps. I just added two more unmetered gigabyte servers with the same config to the pool, please add them to the cn zone too:

https://www.ntppool.org/scores/62.210.205.24
https://www.ntppool.org/scores/78.46.102.180


#252

@felixonmars That’s a large load. I’m sure everyone is thankful for your support. Are we trying to keep the china pool consistent of servers that are located in china? That is why I purchased the Hong Kong VPS as that was my assumption. I guess if you can take that kind of load, although then even a slight higher offset would be looked beyond in the apparent details of the situation.

Thanks for the help, perhaps ask or someone will add it.

Noah
https://logiplex.net


#253

I guess we are not there yet. I have some small Chinese servers but not at all possible to add to the pool at its current state. I can add some if a server setting to 384Kbps would get only 1~2 Mbps.

I hope we can eventually bring the pool back to maybe 30~40 servers and add more to achieve that…


#254

Right. My VPS can’t handle the current state of the CN zone apparently at 384Kbit from what I am hearing. I’m just hanging low in Asia and HK taking about 3k requests a second/6kpps but I figure it’s a start and if we can get more people to donate a low price like $25 for the year I paid for 500GiByte of bandwidth a month, then eventually maybe these HK servers can be merged to the CN zone. Larger servers are of course much more viable and helpful.

I guess we’ll have to wait and see.


#255

avij, I don’t need to prove anything to anybody. Do you really think I am driven by stealth marketing? I think I really should be out of here then.

I have been involved in .cn zone bootstrap since its initial effort in 2016 and it became disheartening to see how after initial pileups servers kept falling off and leaving the zone.
cn

Yesterday’s 22 servers have been reduced to 4 this morning. This is understandable, because nothing has fundamentally changed since 2016.

Whom could I ask to remove 85.199.214.100 and 85.199.214.101 from .cn? They are still being rotated in.
Edited: OK, all my servers are out of cn. I hope it will work out for everyone else!

Thanks
Leo


#256

I want to add my two cents here.

  1. The “Net Speed” setting is misleading.

I took me quite some time digging in the forums to understand it. But any newcomers, especially those with few time (in big companies, etc) or language barrier, will just set it to their servers’ wire speed and afterwards either struggle with the low score (and blame the monitoring system) or broken server networking (due to too much traffic). This will eventually make them decide to remove the server and never come back again.

I would suggest to use a different name here. Like a “Weight”, “Relative Bandwidth” or whatever that makes more sense. I am not a native speaker, so you may find a better one than what I have in mind. Or at least the help text in the right should be changed to explicitly remind them that’s NOT the wire speed.

  1. ntpd is broken on high load.

This is actually hard to figure out and I struggled for quite some time. Please see the following chart:

Before the red bar my server was having a broken firewall setup. And before the green bar, it is a single ntpd instance running with iptables notrack. The ntpd instance drops so many packets that the monitoring ones are dropped from time to time. For details please refer to my other thread: Dropped packets

Between the green and yellow bars it is running with rsntp, which unfortunately resulted in blank lines in the csv log and doomed the score. After the yellow bar it is running with chrony and never had a problem again.

I’m curious if the other 1Gbps server admins are aware of the issue at all, since this can easily be treated as a monitoring system problem as well (because the networking is fine, a few queries to the server will work mostly, and traceroutes to the monitoring station are okay in both direction). I will suggest to try chrony if your server is in the same dilemma.

If later this turns out to be a common problem, I would suggest changes on the software recommendation pages.

  1. Tolerant of broken net speed setting

I believe @LeoBodnar and everyone else in the thread are willing to help with the issues here, and most of us now have a good understanding of what’s going on in the China pool. However, there are many server admins who are either unaware of the forum, don’t have time to, or just didn’t understand. For a volunteer project I think it is important to make the system more tolerant with the fact that many servers are setting a wrong net speed. This is still the case even if the “net speed” name was changed.

For these broken settings, would it be a good idea if the admins go ahead and just lower the net speed setting for them? The issues can easily be identified if the server keeps going down right after getting a score of 10. I know some server admins may want to keep a lower load so increasing the limit would be very inappropriate, but I don’t see a problem the opposite way. If this turns out to be effective, we can even automate it.

In case it’s a human resource problem, I can offer some help here.


#257

Sure is, and I agree.

Had no idea but I don’t doubt it at all. It’s 2019 and ntpd is not even SMP aware afaik.

I can’t think of a way it would be possible to predict and use precise bandwidth measurements with the architecture of the Internet.

Noah
https://logiplex.net


#258

The problem here is that depending on the number of servers in the pool, the number of requests that you get will vary wildly. It would need to be raised again when/if the situation gets better. Not offering a solution here, just a remark.


#259

I see. Something like an admin-only (or adjust automatically by the monitoring system) degrade level might help here, but that certainly involves some development…

Or maybe the “simple” way: An email sent to the server admin with something like “Our monitoring system indicates that your server is overloaded for quite some time (maybe a fixed number, like 7 days). We have thus lowered its net speed setting to improve the service quality. Please try to increase it again when appropriate.”


#260

Well, that is one way of reducing traffic to your servers. Let’s see how this move affects your score and your concerns regarding the monitoring server.


#261

When you create scenarios for pleasing the monitor consider the following facts.

  1. ntppool is not in control of total combined number of user requests R coming into specific zone following the DNS lookups
  2. ntppool is not in control of total combined capacity C of servers in the zone
  3. if for a specific zone R > C the packets will be lost. This includes both: user requests and monitoring packets
  4. If for some zone condition 3 above is true, the most efficient allocation of user request density must be in proportion to server’s share of total bandwidth. Efficient here means that total number of dropped requests is minimal (but not zero.)
  5. In such case all servers will have to drop packets, including monitoring ones.

To make allocation work for a zone for which condition 3 above is true:

  1. Server bandwidth has to be accurately measured or declared. Any fiddling with capacity setting will increase total number of dropped packets.
  2. There must be no penalty for a server dropping incoming packets - monitoring or user ones
  3. Monitor should not remove a server from the pool unless it has failed catastrophically. Server removal will cause all of its allocated packets to be lost - there is no spare capacity to reallocate them to others

The only way of making this work for a zone with R > C is to fix the current monitoring system. Even just turning it off for such a zone is a massive step forward.

Leo


#262

I’d have a few ideas as well, but maybe we should try to reach a consensus about a few requirements before diving into implementation details.

The first one is: Are we allowed to drop requests (by directing it to a server which can’t handle the traffic, perhaps in proportion of their bandwidth setting) if we can’t find a suitable server for responding to those, or should we try harder to find a NTP server for such requests, even if it means we would need to query a server that is far away from the client? I’d lean towards the latter myself.


#263

Since the pool only have tens of servers in China, I think we should ping those server more frequent than we do. Maybe once per miniute? So that we can better profile that server.

And we definatly need a monitor server in China asap. I just check some university’s servers, some of them only response to requests from within China (like s1d.time.edu.cn 4/4 success with ntpdate from Shenzhen, 0/4 from San Jose,CA) . And some response to all request but their return packets were having troubles crossing the wall(eg. s2f.time.edu.cn https://www.pool.ntp.org/scores/202.112.29.82). I think this is a common issue.

My server https://www.pool.ntp.org/scores/120.24.166.46 was not overload for quite some time, but still not able to go above 10. It is hosted by aliyun, suppose to have very nice connection, but the wall is dropping my responses.


#264

one more server for the china zone:
https://www.ntppool.org/scores/116.203.67.76
https://www.ntppool.org/scores/2a01:4f8:1c1c:8038::1

@felixonmars
Yes this is a human resource Problem, like with almost any open Source Project. If you want to contribute, feel free to leave a pull request: https://github.com/abh/ntppool

The Project of adding multiple Monitoring Nodes is an ongoing Project, the beta site had this feature a couple of months, but apparently some more work is needed.


#265

Hi,

two more server for the china zone (fresh installed).
IPv4 & IPv6
Located in europe Paris

https://www.ntppool.org/scores/51.15.241.78
https://www.ntppool.org/scores/2001:bc8:4400:2c00::20:91f

edit: i have removed the server which was in amsterdam. The score was too low all the time.


Score/network woes
#266

Would it be against moral to take all china subzones like hk and merge them to cn due to emergency and maybe start a new thread about anyone with big steel anywhere that wants to help. There is so much fiber these days even my $25 a year hk vps I purchased last night which is openvz of all things chimed in at .007 offset in the USA. Maybe this is an option at least temporarily.

Noah
https://logiplex.net