The issue of NTP requests exceeding bandwidth load

[NTPman]

@umike

I work in a small ISP. For some time we registered a pool zone on our DNS-servers

Of course this is always doable. If the domain name of your small ISP is small-isp.ru then you can configure the clients you distribute with configuration option in the NTP service pointing to ntp.small-isp.ru and you can add the following DNS entry:

ntp.small-isp.ru. IN CNAME 2.eu.pool.ntp.org.

And it is very flexible as it is easy to change where the hardcoded name is pointing to. However, the domain small-isp.ru should always be reachable for the clients to get time synchronization.

[vlad.bfly]

Found this thread by googling. I’m one of those few Russian NTP servers left, and I must admit that I’m having trouble keeping the server up because of the load exceeding my capacity. I’ve been a part of the project for over a decade, and never experienced any issues until recently.

It all began in October, with some mysterious random Internet outages every few hours each taking usually up to ten minutes. I blamed my ISP at first, until their technician told me that during those outages my inbound traffic was maxing out my 100Mbps bandwidth, so their shaper automatically throttled my traffic. The quick research indicated that it was NTP traffic. Normally not more than a few thousand requests per minute, it peaked to half a million and more every once in a while, completely congesting my traffic.

I took my server into monitoring mode for a day, and the situation gradually recovered. However, switching back to operational mode, even with the smallest bandwidth 512Kbit, almost immediately brought the the outages back. I’m still trying to figure out a solution, but it seems like a we are dealing with a snowball effect, when the excessive load makes more and more servers quit, thus even further increasing the load on the remaining ones.

[avij]

Hi @vlad.bfly, let’s continue this discussion in the other topic, as this topic was originally about a slightly different issue in China.

[davehart]

@umike

ntp DDoS reflection attacks, etc

I very much doubt that is an issue today. ntpd’s stable release removed the massive multipler reflector functionality (ntpdc -c monlist) in 2014 or 2015. It was removed from the test releases (ntp-dev) in 2009 or 2010. The reflection DDoS wave in 2015 caused many firewalls to reject all NTP traffic, or all traffic except client (mode 3) and server (mode 4), when all that was needed was to block all ntpdc (mode 7) or specifically mode 7 monlist requests (a particular hex operation code value at the correct offset of the binary mode 7 protocol). Many of those blocks were never removed, causing ongoing carnage for NTP clients and servers for years.

It’s sad the ntp.org release management chose to go 5 years between stable releases despite the potential for DDoS around monlist being understood. I still have major problems with the way release management is being handled, but this isn’t the place to gripe about that.

[kkursor]

@vlad.bfly

Found this thread by googling. I’m one of those few Russian NTP servers left, and I must admit that I’m having trouble keeping the server up because of the load exceeding my capacity. I’ve been a part of the project for over a decade, and never experienced any issues until recently.

It all began in October, with some mysterious random Internet outages every few hours each taking usually up to ten minutes. I blamed my ISP at first, until their technician told me that during those outages my inbound traffic was maxing out my 100Mbps bandwidth, so their shaper automatically throttled my traffic. The quick research indicated that it was NTP traffic. Normally not more than a few thousand requests per minute, it peaked to half a million and more every once in a while, completely congesting my traffic.

I took my server into monitoring mode for a day, and the situation gradually recovered. However, switching back to operational mode, even with the smallest bandwidth 512Kbit, almost immediately brought the the outages back. I’m still trying to figure out a solution, but it seems like a we are dealing with a snowball effect, when the excessive load makes more and more servers quit, thus even further increasing the load on the remaining ones.

Exactly the same. End of October, too.
I’ve rate-limited NTP traffic at the router. It’s better than nothing.

[Bas]

Are you running NTPD or Chrony?

As Chrony is pretty good at limiting traffic to abusive IP’s.

Chrony can keep a log with IP’s that come to often and then stops responding.

I could be wrong, but I do not believe NTPD can.

You may want to switch to encounter this abuse.

Also, what router/connection do you use? I forgot…sorry.

[davehart]

@Bas

Are you running NTPD or Chrony?

As Chrony is pretty good at limiting traffic to abusive IP’s.

As is ntpd, as I pointed out to here recently including a link to the relevant documentation and an example discard minimum directive for ntp.conf. Search the forum for “discard minimum” if you have forgotten. Otherwise, I’m left with the impression you’re intentionally spreading misinformation about ntpd in a misguided effort to promote Chrony. Chrony has lots of other differences with ntpd, there’s no need to stretch the truth to find reasons one might choose it, such as the fact it’s the default in popular Linux distributions.

@Bas

Chrony can keep a log with IP’s that come to often and then stops responding.

I could be wrong, but I do not believe NTPD can.

On the ntpd server, as I’ve also previously pointed out here recently, one can use:

$ ntpq -c "mrulist sort=avgint" | less

The first entries are the ones hitting the server most frequently. There are knobs available via ntp.conf to configure the size of the Most Recently Used list ntpd keeps to enforce rate limiting. This ntpq query displays that list. Without the sort parameter, the list is most recent first.

[Bas]

Nobody asked you Dave.

Chrony keeps a log of abusive users when you enable it.
And keep is as long as you want.

Got nothing with your ntpq command.

Chronyc command prompt is far more advanced.

As such NTPD is NOT my favorite timeserver, far to complicated to get information.

NTPD is old and outdated.

[davehart]

@Bas

Nobody asked you Dave.

This is bullshit, as well as being rude. You claimed ntpd didn’t have a capability and noted you could be wrong. I pointed out a similar capability (not logging, but querying recent). Suggesting I was wrong to respond is just disrespectful of the intelligence of every reader here, and a sign of why I claim your posts can be very similar to trolling.

@Bas

Chrony keeps a log of abusive users when you enable it.
And keep is as long as you want.

This is repetitive, you’ve already stated that and I didn’t contradict you in my reply. Do you just like to make other people read your repetitive comments more than once?

@Bas

Got nothing with your ntpq command.

I wouldn’t expect you to, given you run Chrony. If you ran that command on a ntpd host, or pointing to one that allows remote ntpq queries by tacking on its DNS or IP at the end of the command, you’d have seen its list of most active recent clients.

@Bas

Chronyc command prompt is far more advanced.

As such NTPD is NOT my favorite timeserver, far to complicated to get information.

Repetitive. We all have heard you sing Chrony’s praises and (often misleadingly) run down ntpd before.

@Bas

NTPD is old and outdated.

I’m not sure your opinion on software you don’t use carries much weight. It’s old, as the original NTP server and client, but development is ongoing, though it did a break from 2019 until 2023 when I became active in its development again after demands on my time were reduced.

[davehart]

@kkursor

AFAIK our communication authority has warned Cloudflare several times for local law violation and it may be slowed down/banned/experience other problems.

You should be able to see if in fact cloudflare NTP servers are down in Russia with a few checks.

First off, does nslookup time.cloudflare.com. return any IP addresses for you?

If you do get IP addresses, try using sntp or ntpdate from the NTP distribution to try using time.cloudflare.com:

ntpdate -d time.cloudflare.com

or add them to your NTP server’s configuration with:

server time.cloudflare.com iburst

and then take a look at its status and see if it’s serving you time.

That would be a major help in determining if the problem is some sort of flooding due to abuse/bad client or if it’s due to Cloudflare’s Russian NTP service going down shifting a much higher burden on the rest of the pool’s Russian servers.

[davehart]

@umike

and what is then proposed for client settings? Specify servers of another zone? I think that equepment buyers will be surprised and will not see any logic in this.

The stated policy is for vendors of equipment/software using the pool hardcoded or defaulting to using the pool to obtain a vendor zone from the pool, like ubuntu.pool.ntp.org. Yes, there are many who ignore this, and yes, people have complained about waiting ages for vendor zones to be approved, and I don’t know how much the latter is responsible for the former, but it’s still the advice I’d recommend as it leaves open the possibility to handle a vendor zone differently if it turns out their clients are misbehaving somehow.

Unfortunately that rules out using continent zones as @PoolMUC is suggesting is wise with the current pool DNS implementation. Or at least I couldn’t figure out how to query a vendor+continent zone, perhaps someone affiliated with the pool will speak up here.

[Knot3n]

If you continue to behave like this, especially since you’ve already drawn attention to yourself multiple times, you’ll need to take a break.

[davehart]

@davehart

First off, does nslookup time.cloudflare.com. return any IP addresses for you?

If you do get IP addresses, try using sntp or ntpdate from the NTP distribution to try using time.cloudflare.com:

ntpdate -d time.cloudflare.com

or add them to your NTP server’s configuration with:

server time.cloudflare.com iburst

and then take a look at its status and see if it’s serving you time.

No replies. It doesn’t need to be @kkursor, but would someone in Russia please check if Cloudflare servers are working from within Russia? They are included in *.ru.pool.ntp.org but keep in mind with their IP addresses being anycast from many different data centers, just because monitors outside Russia say it’s working and it is in the zone doesn’t mean it’s working for clients inside Russia. My hunch is it does indeed work inside Russia and that’s letting clients using *.ru.pool.ntp.org continue to work despite what I suspect is a malicious flood of NTP queries to that zone, but it would be groovy to know one way or the other.

If you have access to ntpdate, try:

nslookup time.cloudflare.com.

Does the response include the IP addresses below? Then please try:

ntpdate -d 162.159.200.1
ntpdate -d 162.159.200.123

and if you have IPv6, also:

ntpdate -d 2606:4700:f1::1
ntpdate -d 2606:4700:f1::123

The output should make it clear if you’re getting responses and how close they suggest your system clock is.

If you don’t have access to ntpdate, you can test by adding those addresses to your NTP server configuration and looking at the status after a few minutes to see if you’re getting useful responses.

[timz]

Tested using two different ISP. No one problem detected.

nslookup time.cloudflare.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   time.cloudflare.com
Address: 162.159.200.1
Name:   time.cloudflare.com
Address: 162.159.200.123
Name:   time.cloudflare.com
Address: 2606:4700:f1::123
Name:   time.cloudflare.com
Address: 2606:4700:f1::1

ntpdate -uqd 162.159.200.1
ntpdig: querying 162.159.200.1 (162.159.200.1)
org t1: eaed561c.7dd6f000 rec t2: eaed561c.841cdd9b
xmt t3: eaed561c.84b6ea6c dst t4: eaed561c.8dce4800
org t1: 1732433820.491561 rec t2: 1732433820.516066
xmt t3: 1732433820.518416 dst t4: 1732433820.553929
rec-org t21: 0.024505  xmt-dst t34: -0.035513
2024-11-24 14:37:00.518416 (+0700) -0.005504 +/- 0.030010 162.159.200.1 s3 no-leap

ntpdate -uqd 162.159.200.123
ntpdig: querying 162.159.200.123 (162.159.200.123)
org t1: eaed5678.092aa000 rec t2: eaed5678.0f5f302b
xmt t3: eaed5678.0f62c520 dst t4: eaed5678.16127000
org t1: 1732433912.035807 rec t2: 1732433912.060046
xmt t3: 1732433912.060101 dst t4: 1732433912.086219
rec-org t21: 0.024240  xmt-dst t34: -0.026118
2024-11-24 14:38:32.60101 (+0700) -0.000939 +/- 0.025179 162.159.200.123 s3 no-leap

MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
...
^- 162.159.200.123               3  10   377   818   +115us[ +129us] +/-   40ms
^- 162.159.200.1                 3  10   377   20m   -108us[  -87us] +/-   40ms
...

[kkursor]

@davehart

ntpdate -d 162.159.200.1

No problems.

kkursor@dot:~$ nslookup time.cloudflare.com.
Server: 192.168.14.1
Address: 192.168.14.1#53

Non-authoritative answer:
Name: time.cloudflare.com
Address: 162.159.200.1
Name: time.cloudflare.com
Address: 162.159.200.123
Name: time.cloudflare.com
Address: 2606:4700:f1::123
Name: time.cloudflare.com
Address: 2606:4700:f1::1
kkursor@dot:~$ /sbin/ntpdate -q 162.159.200.1
2024-11-24 11:00:48.595625 (+0300) +0.001640 +/- 0.001534 162.159.200.1 s3 no-leap
kkursor@dot:~$ /sbin/ntpdate -q 162.159.200.123
2024-11-24 11:00:55.581503 (+0300) +0.001635 +/- 0.001247 162.159.200.123 s3 no-leap