NTP at NIST Boulder has lost power

[marco.davids]

Interesting read for the weekend:

jeff.sherman@nist.gov: Dec 19 05:18PM -0800

Dear colleagues,

In short, the atomic ensemble time scale at our Boulder campus has failed
due to a prolonged utility power outage. One impact is that the Boulder
Internet Time Services no longer have an accurate time reference. At time
of writing the Boulder servers are still available due a standby power
generator, but I will attempt to disable them to avoid disseminating
incorrect time.

The affected servers are:
time-a-b.nist.gov
time-b-b.nist.gov
time-c-b.nist.gov
time-d-b.nist.gov
time-e-b.nist.gov
ntp-b.nist.gov (authenticated NTP)

No time to repair estimate is available until we regain staff access and
power. Efforts are currently focused on obtaining an alternate source of
power so the hydrogen maser clocks survive beyond their battery backups.

More details follow.

Full message here: https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/o0dDDcr1a8I

[elvisimprsntr]

Still using one of the affected Boulder servers. The other two are out of Fort Collins.

Demonstrates the benefits of using a geographically diverse time.nist.gov pool vs configuring specific servers.

[2.8.1-RELEASE][root@pfSense]/root: ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
oGPS_NMEA(0)     .GPS.            0 l    1    8  377    0.000   -0.001   0.002
 time.nist.gov   .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+time-b-b.nist.g .NIST.           1 u   66   64  377   50.636   +0.536   0.487
+time-e-wwv.nist .NIST.           1 u   37   64  377   49.767   +0.638   0.275
+time-a-wwv.nist .NIST.           1 u   31   64  377   49.941   +0.425   0.319

[stevesommars]

Also consider using multiple time providers, not just NIST.

In recent months NIST disabled their NTP servers at the University of Colorado. Gaithersburg, Maryland suffered a partial outage.

[Bas]

You have a GPS, so no need to have more then a few servers, typical spread around, not just 1 service.
Never good idea to rely on 1 service alone. I agree with Steve on this.

[elvisimprsntr]

Update on Boulder, CO.

On Dec 21, 2025, at 2:30 PM, ‘jeff.s…@nist.gov’ via Internet-time-service Internet-time-service@list.nist.gov wrote:

Dear colleagues,

Utility power was recently restored to the NIST Boulder campus. Assessment and repair activity is in progress, but I want to give a brief status update regarding Internet Time Services on the NIST Boulder campus. As usual, status notes per-server will be manually updated here:

https://tf.nist.gov/tf-cgi/servers.cgi

Clocks and time transfer services operated from the NIST WWV/Ft. Collins and Gaithersburg, MD campuses are independent and were unaffected throughout.

Soon after the last notice, NIST facilities staff stationed on-site started a diesel generator held in reserve and activated a power transfer switch positioned to supply “second backup” power to the affected laboratory. The period without ac power (due to automatic “first backup” generator failure after 2 days of continuous operation) was about 2 hours. However, large battery banks kept all clocks and most measurement and distribution chains powered throughout. Additional quick action by NIST facility staff secured temperature control for the most sensitive clocks. We regained some monitoring ability showing that the disseminated UTC(NIST) signal likely did not deviate by more than 5 us (five millionths of a second) and appeared stable. Knowing this, I decided to keep the Boulder Internet Time Servers active until we lost monitoring or some other event caused the time scale deviation to increase significantly.

To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay.

NIST provides high-precision time transfer by other service arrangements; some direct fiber-optic links were affected and users will be contacted separately. However, the most popular method based on common-view time transfer using GPS satellites as “transfer standards” seamlessly transitioned to using the clocks at NIST’s WWV/Ft. Collins campus as a reference standard. This design feature mitigated the impact to many users of the high-precision time signal.

Best wishes,

-Jeff Sherman

[ask]

Demonstrates the benefits of using a geographically diverse time.nist.gov pool vs configuring specific servers.

I think also demonstrates some of the benefits of our extremely decentralized approach with the NTP Pool.

[elvisimprsntr]

I thought I read, if you are running a NTP sever connected to a disciplined stratum 0 source, one should use other disciplined and traceable NTP sources as fall back. I believe NTPPOOL weights availability and global diversity more than precision.

I added two stratum 1 USNO servers in addition to the time.nist.gov pool to my GPD+PPS disciplined NTP server. Yes, I fall within the guidelines for use of USNO servers.

[2.8.1-RELEASE][root@pfSense]/root: ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
o127.127.20.0    .GPS.            0 l    1    8  377    0.000   +0.001   0.002
 time.nist.gov   .POOL.          16 p    -   64    0    0.000   +0.000   0.000
+192.5.41.40     .PHC.            1 u   61   64  377   41.609   +0.707   0.935
+192.5.41.41     .PHC.            1 u   40   64  377   41.247   +0.713   1.713
+132.163.96.1    .NIST.           1 u   47   64  377   50.649   +0.610   0.456
+132.163.96.4    .NIST.           1 u   48   64  377   50.502   +0.486   0.459