# Leap second 2017 status

**URL:** <https://community.ntppool.org/t/leap-second-2017-status/59>\
**Category:** Uncategorized\
**Created:** [December 31, 2016, 10:04pm UTC](https://community.ntppool.org/t/leap-second-2017-status/59 "2016-12-31T22:04:55Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\
**Post date:** [December 31, 2016, 10:04pm UTC](https://community.ntppool.org/t/leap-second-2017-status/59/1 "2016-12-31T22:04:56Z")

</div>

As of this writing (about 3 hours before the leap second) about 3463 servers in the pool system are announcing the leap second. 933 are not.

There are also 10 servers that have the “unsynchronized” bit set.

Filtering out the servers that monitoring has deemed unhealthy, the numbers are 3344, 568 and 2.

---

<div class="post-metadata">

**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\
**Post date:** [January 1, 2017, 12:45am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/2 "2017-01-01T00:45:08Z")

</div>

Now 40 minutes after the leap second, 442 servers in the pool are off by a second. 384 of them are still announced in the pool system (even being a second off it takes a couple measurements to get kicked out; I’ve made a todo to make the system more prompt about that).

Amazingly there are also 4 servers that are off by a second in the wrong direction (like they applied the leap second twice).

It’s pretty amazing how poorly this works, decades later.

---

<div class="post-metadata">

**Author:** ![josephb](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/josephb/32/7_2.png) [@josephb](https://community.ntppool.org/u/josephb)\
**Post date:** [January 1, 2017, 3:39am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/3 "2017-01-01T03:39:46Z")

</div>

Thanks for sharing the stats.

Those 4 must be quite special flowers 🙂

---

<div class="post-metadata">

**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\
**Post date:** [January 1, 2017, 6:03am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/4 "2017-01-01T06:03:27Z")

</div>

Last update on this, at least for today:

It dropped off pretty fast:

A little later (01:00-02:00) it was a little over 220 servers off by a second (plus/minus 50ms).  
02-03: 164 servers.  
03-04: 112.  
04-05: 87  
05-06 (now): 66

There are a few dozen more that are off by a significant amount but less than a second. At a glance they look like slowly recovering.

All are excluded from the DNS records, but old clients might be using them of course.

I didn’t look super carefully, but I also saw a few servers that were correct at midnight UTC, but were off by a second for a brief period hours later.

If there’s any conclusion then I think it is that handling leap seconds is still pretty much broken. Obviously not all operators of servers in the NTP Pool take the same care or interest or have the same knowledge of all this, but my expectation is that the overall NTP installations aren’t doing better than this.

It’d be interesting to break this down by server software and versions, but we don’t have that information in the NTP Pool system.

---

<div class="post-metadata">

**Author:** ![alica](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/alica/32/337_2.png) [@alica](https://community.ntppool.org/u/alica)\
**Post date:** [January 1, 2017, 6:34am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/5 "2017-01-01T06:34:34Z")

</div>

Last night around 9pm in Taiwan (12/31 1pm UTC) I checked my server’s backup time sources, and found that Korea and Philippines government-running standard time servers have not published leap second warning, while standard time servers from Taiwan/Hong Kong/Japan/Indonesia published warnings. I did not check further (during the leap second occurrence) and no screenshots were available, though. After all, their time is correct by now. 👿

---

<div class="post-metadata">

**Author:** ![ghane0](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ghane0/32/29_2.png) [@ghane0](https://community.ntppool.org/u/ghane0)\
**Post date:** [January 1, 2017, 11:26am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/6 "2017-01-01T11:26:45Z")

</div>

> [@ask](#):
>
> If there’s any conclusion then I think it is that handling leap seconds is still pretty much broken. Obviously not all operators of servers in the NTP Pool take the same care or interest or have the same knowledge of all this, but my expectation is that the overall NTP installations aren’t doing better than this.

Could this be due to the servers syncing to an upstream server that is implementing leap smearing?

See: [http://www.n5tnl.com/time/leap\_2016/index.html](http://www.n5tnl.com/time/leap_2016/index.html)

---

<div class="post-metadata">

**Author:** ![mlichvar](https://avatars.discourse-cdn.com/v4/letter/m/e79b87/32.png) [@mlichvar](https://community.ntppool.org/u/mlichvar)\
**Post date:** [January 1, 2017, 3:07pm UTC](https://community.ntppool.org/t/leap-second-2017-status/59/7 "2017-01-01T15:07:58Z")

</div>

I was monitoring a large part of the pool servers during the leap and I was curious to see how many servers that didn’t leap correctly share the same source. I tried to prepare a graph of synchronization from reference IDs from packets collected immediately after the leap.

[https://mlichvar.fedorapeople.org/leap2016/badleap\_refids.png](https://mlichvar.fedorapeople.org/leap2016/badleap_refids.png)

There are some clusters of servers that share the same source, but it seems the problem is spread quite uniformly. I suspect the main problem is configuration using a small number of sources, which can’t outvote the bad servers or reference clocks. About a quarter of the servers seems to be running openntpd, which doesn’t support leap seconds yet.

The largest cluster is synchronised to refid 7F7F01\* (aka local driver). The recommendations for pool servers say the local driver shouldn’t be used, so maybe it would be good to send an email to the owners of the servers that did?

---

<div class="post-metadata">

**Author:** ![alica](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/alica/32/337_2.png) [@alica](https://community.ntppool.org/u/alica)\
**Post date:** [January 1, 2017, 4:50pm UTC](https://community.ntppool.org/t/leap-second-2017-status/59/8 "2017-01-01T16:50:17Z")

</div>

> [@mlichvar](#):
>
> About a quarter of the servers seems to be running openntpd, which doesn’t support leap seconds yet.

Why should we tolerate this [defective-by-design](http://undeadly.org/cgi?action=article&sid=20150628132834) software live inside the pool???

---

<div class="post-metadata">

**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\
**Post date:** [January 2, 2017, 2:46am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/9 "2017-01-02T02:46:02Z")

</div>

Alica, I didn’t realize openntpd just ignores the leap second. Those probably aren’t appropriate for the pool then, at least not in regions of the world with plenty servers.

That being said, it’s not super clear what to do to improve the situation. Exclude servers if they fail a leap second? It’s somewhat random depending on which upstream servers they have at the moment and lots can change until the next leap second (might be in 6 months, might be in 3 years).

---

<div class="post-metadata">

**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\
**Post date:** [January 2, 2017, 2:47am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/10 "2017-01-02T02:47:43Z")

</div>

> [@ghane0](#):
>
> Could this be due to the servers syncing to an upstream server that is implementing leap smearing?

No, those were pretty easy to spot. There [were 9](https://community.ntppool.org/t/google-public-ntp/44/12) in the system (that got kicked out after a few hours).

---

<div class="post-metadata">

**Author:** ![mnordhoff](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/mnordhoff/32/1470_2.png) [@mnordhoff](https://community.ntppool.org/u/mnordhoff)\
**Post date:** [January 2, 2017, 2:53am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/11 "2017-01-02T02:53:18Z")

</div>

What if the monitoring code was aware of leap seconds? Something like:

- If there’s a leap second today `&&` the server is not announcing it:
  - `$step -= 0.2;`

If a server is consistently unaware of the leap second, and my math is right, that should dock about 20 points over the course of the day, disabling it around 12:00, while it can continue to contribute for the rest of the year.

---

<div class="post-metadata">

**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\
**Post date:** [January 2, 2017, 3:07am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/12 "2017-01-02T03:07:00Z")

</div>

> [@mnordhoff](#):
>
> What if the monitoring code was aware of leap seconds? Something like: …

For NTP clients not handing out these servers in the last day before the leap is likely too late. Many ntpd’s will have been running for weeks or months already.

SNTP clients don’t care about the leap second, but it’d potentially protect them from talking to a confused server just after the leap second (until the monitoring system has kicked those out anyway).

Now, I’m not sure what the overlap between “didn’t announce the leap second” and “were off by a second after the event” were (it’s in the data though if anyone is up for trying to figure that out). I’m sure it’s less than 100%.

The downside to implementing something like that is that the pool system now has to get all this correct, beyond just having the right time. It wouldn’t be a lot of code to implement, but it doesn’t “feel good”. It’s something that’s hard to test and has to work – or the whole system would go kaboom[1].

Since it won’t help NTP clients (vs SNTP) it doesn’t seem worth the complexity.

This leap second I was around, had time and good internet connection, but generally I optimize for “will work without me sitting paying attention”. The NTP Pool doesn’t exactly have a NOC or 24/7 staff. Or staff. 🙂

I think there are other things that are more likely to be beneficial, in particular things that are actionable weeks or months in advance. For example we could track the refid/upstream of each server, learn which ones mess up the leap second and warn operators to not use those as upstream servers. Maybe.

[1] (Which I suppose is the story of leap seconds).

---

<div class="post-metadata">

**Author:** ![mnordhoff](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/mnordhoff/32/1470_2.png) [@mnordhoff](https://community.ntppool.org/u/mnordhoff)\
**Post date:** [January 2, 2017, 3:18am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/13 "2017-01-02T03:18:39Z")

</div>

> [@ask](#):
>
> For NTP clients not handing out these servers in the last day before the leap is likely too late. Many ntpd’s will have been running for weeks or months already.
> 
> SNTP clients don’t care about the leap second, but it’d potentially protect them from talking to a confused server just after the leap second (until the monitoring system has kicked those out anyway).

Hrm, you’re right.

> [@ask](#):
>
> The downside to implementing something like that is that the pool system now has to get all this correct, beyond just having the right time. It wouldn’t be a lot of code to implement, but it doesn’t “feel good”. It’s something that’s hard to test and has to work – or the whole system would go kaboom[1].
> 
> Since it won’t help NTP clients (vs SNTP) it doesn’t seem worth the complexity.

Agreed. ☹

> [@ask](#):
>
> This leap second I was around, had time and good internet connection, but generally I optimize for “will work without me sitting paying attention”. The NTP Pool doesn’t exactly have a NOC or 24/7 staff. Or staff. 🙂

It would be pretty much set-it-and-forget-it. Update `leap-seconds.list` occasionally, by hand or by cron. Then cross your fingers and hope it doesn’t go disastrously wrong. 😄

> [@ask](#):
>
> For example we could track the refid/upstream of each server, learn which ones mess up the leap second and warn operators to not use those as upstream servers. Maybe.

Mm. That sounds _very_ complex for something with limited accuracy, though.

Edit: And there’s no way to predict operators who use problematic upstreams but configure `leap-seconds.list` by hand. Or only use 49% problematic upstreams, so they’d still be alright.

---

<div class="post-metadata">

**Author:** ![mnordhoff](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/mnordhoff/32/1470_2.png) [@mnordhoff](https://community.ntppool.org/u/mnordhoff)\
**Post date:** [January 2, 2017, 4:16am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/14 "2017-01-02T04:16:06Z")

</div>

@ask

On the other side of the leap second, what to do about servers that are now 1 second off? Under the current scheme, it can take them about 100 minutes [edit: or about 110] to leave the pool. For example:

```
1483233900,"2017-01-01 01:25:00",0.987508058547974,-2,8.9,0
1483232417,"2017-01-01 01:00:17",0.991002321243286,-2,11.4,0
1483231040,"2017-01-01 00:37:20",0.99502968788147,-2,14.1,0
1483229629,"2017-01-01 00:13:49",0.999479174613953,-2,17,0
1483228172,"2016-12-31 23:49:32",-0.00108015537261963,1,20,0 

```

From [`Monitor.pm`](https://github.com/abh/ntppool/blob/8f7cf2d46063874f78ae6db8b647124932c61c79/lib/NTPPool/Control/Monitor.pm#L118):

```
           if ($offset_abs > 3 or $status->{stratum} >= 8) {
               $step = -4;
               # [...]
           }
           elsif ($offset_abs > 0.75) {
               $step = -2;
           }

```

What if it was changed to take 5-10 points if they’re \> 0.9 seconds off? Or 0.9 - 1.1?

That would be a simple change, and beneficial to SNTP clients from around 00:30-01:30 after a leap second.

More complicated rules might be better, but that should be good enough. (Only punish ~1.0 second servers on a leap second day? Don’t be so harsh if they already have a low score, so they don’t get to -100 in a few hours?)

Edit: Or have the monitoring system run a special check a few seconds after a leap second, instead of waiting up to ~15 minutes for the normal scheduled check.

Edit (06:46): I unwittingly edited out three of the more important lines of code.

---

<div class="post-metadata">

**Author:** ![Sparks](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@Sparks](https://community.ntppool.org/u/Sparks)\
**Post date:** [January 2, 2017, 4:37am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/15 "2017-01-02T04:37:12Z")

</div>

> [@mnordhoff](#):
>
> Update leap-seconds.list occasionally, by hand or by cron. Then cross your fingers and hope it doesn’t go disastrously wrong. 😄

Having just recently read the [NTP Best Current Practices](https://tools.ietf.org/html/draft-ietf-ntp-bcp-02) document, I have plans to pull in the leap-seconds.list using cron. Does ntpd have to be restarted afterwards or does it occasionally take a peek over there to see if there is anything new?

---

<div class="post-metadata">

**Author:** ![Sparks](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@Sparks](https://community.ntppool.org/u/Sparks)\
**Post date:** [January 2, 2017, 4:38am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/16 "2017-01-02T04:38:11Z")

</div>

> [@ask](#):
>
> about 3463 servers in the pool system are announcing the leap second. 933 are not.

How can you see this?

---

<div class="post-metadata">

**Author:** ![mnordhoff](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/mnordhoff/32/1470_2.png) [@mnordhoff](https://community.ntppool.org/u/mnordhoff)\
**Post date:** [January 2, 2017, 4:39am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/17 "2017-01-02T04:39:13Z")

</div>

> [@Sparks](#):
>
> Does ntpd have to be restarted afterwards or does it occasionally take a peek over there to see if there is anything new?

Modern versions (4.2.8; maybe 4.2.6) check the file automatically. (Once a day, i think; maybe once an hour.)

if 4.2.6 doesn’t do it automatically, you can still use ntpq (maybe ntpdc) to reload it, if you have the control key stuff set up.

---

<div class="post-metadata">

**Author:** ![Sparks](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@Sparks](https://community.ntppool.org/u/Sparks)\
**Post date:** [January 2, 2017, 4:44am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/18 "2017-01-02T04:44:27Z")

</div>

> [@mnordhoff](#):
>
> Modern versions (4.2.8; maybe 4.2.6) check the file automatically. (Once a day, i think; maybe once an hour.)

Okay, looks like I’m using 4.2.6 here (CentOS 7). Is there a way to know whether or not it’s looking? Is it in the logs or is there a command I can run to see?

---

<div class="post-metadata">

**Author:** ![mnordhoff](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/mnordhoff/32/1470_2.png) [@mnordhoff](https://community.ntppool.org/u/mnordhoff)\
**Post date:** [January 2, 2017, 4:48am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/19 "2017-01-02T04:48:25Z")

</div>

> [@Sparks](#):
>
> Okay, looks like I’m using 4.2.6 here (CentOS 7). Is there a way to know whether or not it’s looking? Is it in the logs or is there a command I can run to see?

It’s been years since i ran 4.2.6 with a leap second file, if i ever did…

4.2.8 includes leap file information in `ntpq -c rv` or, more specifically, `ntpq -c "rv 0 leapsec expire"`. Check if 4.2.6p5 does the same?

I _think_ it also logs a message when the file has changed, but i’m not certain.

---

<div class="post-metadata">

**Author:** ![Sparks](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@Sparks](https://community.ntppool.org/u/Sparks)\
**Post date:** [January 2, 2017, 4:52am UTC](https://community.ntppool.org/t/leap-second-2017-status/59/20 "2017-01-02T04:52:20Z")

</div>

> [@mnordhoff](#):
>
> 4.2.8 includes leap file information in ntpq -c rv or, more specifically, ntpq -c “rv 0 leapsec expire”. Check if 4.2.6p5 does the same?

Hmmm, I see “leap\_none”, not sure if that means it’s not expecting a leap second or… ?

```
[root@server ~]# ntpq -c rv
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.6p5@1.2349-o Mon Nov 14 18:25:09 UTC 2016 (1)",
processor="x86_64", system="Linux/4.8.6-x86_64-linode78", leap=00,
stratum=2, precision=-24, rootdelay=1.484, rootdisp=63.680,
refid=200.98.196.212,
reftime=dc145294.b2a902b9 Mon, Jan 2 2017 4:16:20.697,
clock=dc145a48.cf2a1ab9 Mon, Jan 2 2017 4:49:12.809, peer=23189,
tc=10, mintc=3, offset=-3.633, frequency=-7.465, sys_jitter=20.045,
clk_jitter=11.233, clk_wander=1.035
[root@server ~]# ntpq -c "rv 0 leapsec expire"
No system variables returned

```

[Next page](https://community.ntppool.org/t/leap-second-2017-status/59.md?page=2)