FYI. A number of NTP servers began setting leap=1 beginning 2021-11-27 00:00
My local LeoNTP is affected.
1 Like
What firmware version do you have?
Apparently, there was a leap second bug fixed in 1.24: http://www.leontp.com/firmware/
FWIW, from servers I see in my monitoring logs announcing a leap second: 8 are leontp servers, 29 are some other stratum-1 servers (something else than ntpd, chronyd, openntpd), and 18 stratum-2 servers (mostly openntpd), which seem to be accepting the leap from some of the buggy stratum-1 servers.
Isn’t the pool monitoring supposed to remove such servers from the pool? Their scores don’t seem to be impacted.
My LeoNTP runs 1.19, which is the latest version supporting remote monitoring.
I’ve suggested monitoring the leap indicator to Ask, but it hasn’t bubbled to the front of the todo queue.
Note that it has been 256 weeks - 1 day since the last leap second.
I upgraded the LeoNTP to 1.24 and the leap indicator went back to 0.
Great.
It will be interesting to see what happens tomorrow, whether they will be stuck with the leap flag, or they accept the leap second and will serve time off by one second, or something else.
The system does record the leap flag in the monitoring logs. At a glance (the SQL query is a little dumb) there are ~75 servers with leap=1 right now.
As an example from the CSV logs, too:
https://www.ntppool.org/scores/81.169.234.204/log?limit=200&monitor=*
You are right though the system doesn’t do anything with this data. Part was from an assumption that by the time a server shows the wrong leap flag it’s probably too late to do anything about it; but clearly right now that’s not the case.
The other concern I had was (correctly) keeping the system up to date with what the correct leap second schedule is.
What would be the reasonable thing to do in this case? Mark the servers unhealthy and notify the operators that they’re erroneously setting a leap flag?
I think for when there is a leap second, it isn’t practical to tell operators they missed it, right? (Because the spec altows you to only flag the leap second relatively late, if I remember right).
1 Like
I would notify operators of the incorrect flag & immediately drop them out of the pool.
I’ve seen many instances of “false leap second indicators”, usually at the last day of the month. In many cases the server executes the leap second procedure at the last second of the month.
!!! This is happening now (2021-11-30). !!! A number of public NTP servers are setting leap=1. This is a follow-on from the 2021-11-27 problem.
1 Like
Yes, I agree that their score should drop immediately to be removed from the pool.
Today, I see only 4 stratum-1 servers announcing leap. They seem to be ntpd servers. As ntpd accepts leap only on the last day of a month, it is possible that their faulty refclocks were announcing it since the 27th, but only today it was accepted and it spreads over Internet to other servers.
Anything you two, @mlichvar & @stevesommars, agree on so easily I can get behind. 
I think for an implementation plan I will setup a data scheme defining periods where the expected leap state is known. When in those periods the system can change the scoring based on the leap flag (and alert operators accordingly).
I understand the current state of “no leap flag”, so I think the immediate configuration would be (dates inclusive)
From: 2021-07-01
To: 2022-05-31 (?)
State: Leap flag must be 0
Then when the end of June leap is announced we’d add a new entry for the next period. Does that sound right? Is end of May the right time to end the current state?
I only understand how to alert on wrong leap flags; but maybe that’s all that’s reasonable to do anyway?
If a leap flag is announced for end of June 2022, we’d expect people to (correctly) start announcing it June 1st, but responses without the leap flag are expected through the full period.
Is there a point in the last 24 hour period where we should stop using servers that don’t announce the leap? Since most systems that pay attention to the leap flag will be ntp daemons, I suspect by then it’s too late to try to exclude sources that don’t. (And obviously we’d have to go through at least one leap seconds and collect data before doing this).
The upcoming leap second status is announced in early July & January by IERS.
See IERS - IERS News - IERS Bulletin C (leap second announcements) - latest issue
We know the upcoming leap second status at least 5 months in advance.
Looking at BCP223
However, because some older clients may apply it at the
end of the current day, it is RECOMMENDED that NTP servers wait until
the last day of the month before broadcasting leap seconds
Removing the NTP server from the pool on the last day of the month is only a partial solution, because many clients will not do an NTP server reselection. There’s not much more that the NTP pool can do.
So would it be reasonable for the monitoring system to only handle the case of “there’s no leap second announced but the flag is set”?
When a leap second is announced, it sounds like basically anything is acceptable (even if not recommended)?
Nobody knows when the next leap second will occur.
There are predictions that a negative leap second will occur at the end of 2029. [Leap=2].
Maybe for the case “leap section due, but leap indicator=0”, the monitor could send a warning to the server admin. Won’t be able to test for some time.
1 Like
It is now 50 minutes since the 2021-12-01 “false leap second”. a number of servers are still in error by 1 second. Here is one example, chosen at random
(wrong)PHK had a DNS powered leap second information distributor. It might still be updated, haven’t checked, don’t think it’s used much.
There is way way for Autokey (RFC 5906) to distribute leap second information, I think it’s broken, don’t use it.
Receivers for satellite trilateration (GPS) can get information on leap seconds and seem to bungle it.
I can’t say anything good about longwave, FM, ATSC, DVB whatever, or cell tower provided time.
Only Gentoo, Debian, and some derivatives seem to package the tzdb derivative of IERS bulletin C.
Chrony uses a different file which might be more commonly available.
Leap second information doesn’t seem well distributed and might never be.