Collapse of Russia country zone

Okay. What methodology are you using to show the correlation, so I can try to replicate your findings?

Run tcpdump a couple of hours before and after adding a server to the pool. Itā€™d be interesting to see what you find.

RU zone resurrected. I opened the valve at home internet and it survives load correctly.
People in comments blame that Yandexā€™ smart speaker is the cause of NTP flood. Yesterday evening server load drastically reduced. What was the reason? Increase of server power? Fixing of some firmware bug of a widespread device? Maybe both.
Welcome back, ru.pool.ntp.org. I am proud to be part of it again.

10 Likes

Welcome back :slight_smile:
and thanks for the news.

Maybe the ā€œmonitoring onlyā€ server will come back and re-join the pool :slight_smile:

really many NATā€™s use the whole range of ports. For example (as far as I remember) Mikrotik can use ports starting from 1 unless to-ports range not configured. So I think that most of requests with port <1024 can be legitimate. They just come because of NAT. Maybe home soho routers (CPE) behave the same way.

See Figure 8 in this NIST paper for a discussion of NTP client source ports.

1 Like

As someone who is just setting up to serve the same country zone (yes, I responded to the ā€œdistress callā€ā€¦) ā€” mind if I ask, which NTP upstream(s) and which strata are you using?

The VNIIFTRI cluster is down, and while Iā€™m lucky to have direct access to MSK-IX (and thus to the MSK-IXā€™s stratum 1 server), Iā€™d prefer if I could have more ā€œreliableā€ upstreams than just one.

Yes, that was Yandex, who did this: ŠžŠ± ŠøŠ½Ń†ŠøŠ“ŠµŠ½Ń‚Šµ с NTP-сŠµŃ€Š²ŠµŃ€Š°Š¼Šø / Š„Š°Š±Ń€

One should google it. Maybe vniiftri was not ready for a habraeffect.

vniiftri have servers in another zones, see on site. People say that they were removed from DNS, but now some have already started working

You can try CDN servers
server time.google.com noselect
server time.facebook.com
server time.cloudflare.com

Google Public NTP serves leap-smeared time. We use this technology to smoothly handle leap seconds with no disruptive events.
it is not standard so it is not recommended

There are a few comments about Vninftri in telegram nag_public. Iā€™s ā€œnear telecom/ISPā€ talks/flood :slight_smile:

Usually, many countries provide legal time services, typically provided by their navies, observatories or measurement authorities. Among such services, NTP is typically one of them, often public NTP servers. Doesnā€™t Russia?

Those were exactly the VNIIFTRI servers I mentioned (VNIIFTRI is essentially the Russian ā€œNIST Time and Frequency Divisionā€). As of yesterday, the entire cluster was offline.

As a matter of fact, they have just restored that service (half of that cluster is reachable from where my server sits in Moscow) ā€” so the immediate problem is resolved and I have just enough upstreams to abide by the ntppool guidelines.

1 Like

For the record: Š£ŃŠ»ŃƒŠ³Šø | Š¤Š“Š£ŠŸ Š’ŠŠ˜Š˜Š¤Š¢Š Š˜

OT:
NTP.org provides a wiki with Straum 1 and Stratum 2 servers.

https://support.ntp.org/Servers/StratumOneTimeServers
https://support.ntp.org/Servers/StratumTwoTimeServers

1 Like

Nearly twice server in ru zone before the downfall!

1 Like

NTP.org provides a wiki with Straum 1 and Stratum 2 servers

Yeah, this wiki is linked from the NTPPool guidelines, so that was the first place I looked.

Unfortunately, most of (RU) S1 servers in that list are inactive/unreachable. The ones that still work are personal hosts, and all of them are pretty far away from Moscow (judging by ping times), so not an ideal choice.

As Chinese and former employee of Tencent, I must said this post has too many misunderstand/speculation of Chinese ISPs/NTP status.

The main reason why Chinese NTP server canā€™t operate well is ISP bill for business is way more expensive than residential usage (100MBps monthly for example, ~2600 CNY vs 10 CNY).
Also thanks to IANA, only 350 million IPv4 address are assigned to China, meanwhile we have more than 1 billion netizen, so residential ISP canā€™t have static IPv4 address and 99.9% are behind NAT due to lack of IPv4 addresses.

Thus NTP servers are mostly operated by big companies or some government based institutions. Hobbyist NTP must be rich enough to bought private own business line (rich, I mean China is a developing country with median income 33036 CNY yearly, 2023)

Another issue is connectivity to ā€œthe Internetā€.

First of all, please bear in mind geography and physic, monitors mostly operated in US/EU, thatā€™s 10000 KM alway from China, itā€™s will takes 100ms+ by Einstein ā€œSpecial relativityā€, another 60ms by fiber losses, router hop delay etc. RTT should be more than 350ms, for NTP, thatā€™s not good.

Secondly, there is just too many netizen, period.
If you had server in China, during night time, package loss rate will be increase drastic by huge amount of packages, UDP based protocol are first in line drop target of QoS for most of ISPs policy that I known of.

Thirdly, ISPs competition.
Yes. China has three major ISPs and lots of small ISPs, not ā€œONLY one evil government controlled/censoredā€ Internet provider.
The way ISPs keep their customer including but not limited to: deteriorate other ISP package loss, little gifts/discount if you sign long term contract, setup higher fee for interconnect with ISP, cut off some other ISP fiber ā€œby accidentā€.
Oversea Internet package often came from other ISP instead of the ISP that end-user in, so drop others package is quite common if ISP hits high bandwidth usage.

3 Likes

Nearly twice server in ru zone before the downfall!

some of these servers are rented virtual machines, they will be turned off as soon as people get tired of paying for them. The other part are home servers. Letā€™s see how long they will work :slight_smile:

By the way, is it okay to use virtual machines for a time server? I thought that a good server (even no stratum-1) requires real hardware.

2 Likes

Ahh yeah, youmention the 30 VPS earlier. But itā€™s looks good anyway.

And regarding VPS - if you need highly accurate time you wonā€™t use the pool :smiley: You would run in your own network your own NTP or better PTP :slight_smile:

You can have straum 6 server which is more accurate as stratum 1 with piggy hardware :slight_smile:

Sometimes the host have a high frequency drift which canā€™t be compensatet anymore by ntp (>= 500ppm) you need / can try chrony wich will work.

Most of my servers a vps are running fine and they arenā€™t worse than a rented hardware server. But both of them will fail in comparsion to a ntp server with ā€œhardware accelearionā€ like ntp in FPGA.

And as bas often said - you can build your own accurate ntp server for a couplde of bucks with a ex. a Pi and any good GNNS receiver (Garmin, U-Blox or Trimble or ā€¦)

It depends on a lot of unstated details, so itā€™s not really possible to say simply yes or no.

For a pool server, accuracy in the single-digit milliseconds is more than enough to stay in the pool. For a pool monitor, on the other hand, Iā€™d want appreciably less than 1 msec offsets seen locally, and as symmetric path as possible to handoff(s) to other ISPs (AS numbers).

Thereā€™s a previous thread here that discusses the age-old ā€œbare metal only for NTPā€ which has become less mandatory depending on details like type of hypervisor, CPU virtualization support, etc: Cringe at thought of NTP server on a VM?

Would you say itā€™s fair to describe the price difference between business and residential internet pricing in China as ā€œbusiness internet subsidizes home (residential) internetā€?

I presume when you say nearly all home internet is behind NAT, youā€™re talking about an ISPā€™s NAT, so that numerous homes share a single public IPv4 address? For clarity, itā€™s best to refer to that type of NAT as ā€œCGNATā€ (carrier-grade NAT). CGNAT implies no ability to forward NTPā€™s UDP port 123 to a specific device on the home network, so no possibility to host a public NTP server or any other service requiring a specific port or ports.

Regarding most NTP pool monitors being outside China, Iā€™m pretty confident monitors inside China would be welcome. If you know people with systems in Chinese data centers interested in helping out, the IP traffic, CPU load and disk space involved are all very small. [1] You might get them to bring up a pool server first and introduce themselves on this forum. Note I do not represent the pool, Iā€™m just an enthusiastic participant.

I think I recall but canā€™t find right now reading here a message where @Ask mentioned the possibility of donating a bare metal or VM system with good timekeeping (such as a GNSS-connected NTP server or appliance in the same datacenter) running Linux/FreeBSD which he would configure as a monitor. I donā€™t know if he really wants one more server to manage, though. If it were me Iā€™d much prefer someone willing to manage it themselves. The basic requirements are Linux or FreeBSD with 24x7 solid symmetric connectivity and established ability to run Chrony or ntpd showing stable performance. Given advances in x86/x64 hardware virtualization assistance and type 1 hypervisors that use them, while bare metal is always preferable it is possible do manage well with a VM.

[1] Monitor traffic load estimate by Ask in 2018