IPv6 servers' pool monitoring frequency sudden decrease

Pool Development
1

Continuing the discussion from Server monitoring:

Since around two days the frequency of monitoring decreased, but only for certain servers. Here is an example:

image

Here is the log:

ts_epoch,ts,offset,step,score,monitor_id,monitor_name,leap,error
1633242745,"2021-10-03 06:32:25",-0.005832321,1,16,10,"San Jose, CA, US",0,
1633240414,"2021-10-03 05:53:34",-0.005899552,1,15.8,10,"San Jose, CA, US",0,
1633236511,"2021-10-03 04:48:31",-0.005840395,1,15.6,10,"San Jose, CA, US",0,
1633235048,"2021-10-03 04:24:08",-0.004195286,1,15.4,10,"San Jose, CA, US",0,
1633229388,"2021-10-03 02:49:48",-0.005711952,1,15.1,10,"San Jose, CA, US",0,
1633225209,"2021-10-03 01:40:09",-0.005882375,1,14.9,10,"San Jose, CA, US",0,
1633223531,"2021-10-03 01:12:11",-0.005796941,1,14.6,10,"San Jose, CA, US",0,
1633218962,"2021-10-02 23:56:02",-0.005780455,1,14.3,10,"San Jose, CA, US",0,
1633215475,"2021-10-02 22:57:55",-0.005879503,1,14,10,"San Jose, CA, US",0,
1633214438,"2021-10-02 22:40:38",-0.005846472,1,13.7,10,"San Jose, CA, US",0,
1633212277,"2021-10-02 22:04:37",-0.005830691,1,13.3,10,"San Jose, CA, US",0,
1633210387,"2021-10-02 21:33:07",-0.00583087,1,13,10,"San Jose, CA, US",0,
1633204107,"2021-10-02 19:48:27",-0.006514651,1,12.6,10,"San Jose, CA, US",0,
1633197150,"2021-10-02 17:52:30",-0.005900456,1,12.2,10,"San Jose, CA, US",0,
1633189755,"2021-10-02 15:49:15",-0.005810476,1,11.8,10,"San Jose, CA, US",0,
1633187186,"2021-10-02 15:06:26",-0.005783609,1,11.4,10,"San Jose, CA, US",0,
1633182495,"2021-10-02 13:48:15",-0.005666535,1,11,10,"San Jose, CA, US",0,
1633178843,"2021-10-02 12:47:23",-0.005654931,1,10.5,10,"San Jose, CA, US",0,
1633173178,"2021-10-02 11:12:58",-0.005770897,1,10,10,"San Jose, CA, US",0,
1633170620,"2021-10-02 10:30:20",-0.00578989,1,9.4,10,"San Jose, CA, US",0,
1633167291,"2021-10-02 09:34:51",-0.005915668,1,8.9,10,"San Jose, CA, US",0,
1633158495,"2021-10-02 07:08:15",-0.00564104,1,8.3,10,"San Jose, CA, US",0,
1633155596,"2021-10-02 06:19:56",-0.004186487,1,7.7,10,"San Jose, CA, US",0,
1633153428,"2021-10-02 05:43:48",-0.005902204,1,7,10,"San Jose, CA, US",0,
1633147106,"2021-10-02 03:58:26",-0.005840044,1,6.4,10,"San Jose, CA, US",0,
1633140728,"2021-10-02 02:12:08",-0.0058251,1,5.6,10,"San Jose, CA, US",0,
1633136359,"2021-10-02 00:59:19",-0.005704056,1,4.9,10,"San Jose, CA, US",0,
1633130955,"2021-10-01 23:29:15",-0.005657979,1,4.1,10,"San Jose, CA, US",0,
1633125801,"2021-10-01 22:03:21",-0.005721176,1,3.3,10,"San Jose, CA, US",0,
1633124586,"2021-10-01 21:43:06",0,-5,2.4,10,"San Jose, CA, US",,"i/o timeout"
1633121580,"2021-10-01 20:53:00",-0.005916181,1,7.8,10,"San Jose, CA, US",0,
1633115812,"2021-10-01 19:16:52",-0.006733976,1,7.1,10,"San Jose, CA, US",0,
1633112326,"2021-10-01 18:18:46",-0.00712894,1,6.4,10,"San Jose, CA, US",0,
1633108430,"2021-10-01 17:13:50",-0.005635393,1,5.7,10,"San Jose, CA, US",0,
1633103109,"2021-10-01 15:45:09",-0.005551216,1,5,10,"San Jose, CA, US",0,
1633098696,"2021-10-01 14:31:36",0,-5,4.2,10,"San Jose, CA, US",,"i/o timeout"
1633093403,"2021-10-01 13:03:23",-0.004396662,1,9.7,10,"San Jose, CA, US",0,
1633088117,"2021-10-01 11:35:17",-0.005347815,1,9.1,10,"San Jose, CA, US",0,
1633085333,"2021-10-01 10:48:53",-0.005707239,1,8.5,10,"San Jose, CA, US",0,
1633082114,"2021-10-01 09:55:14",-0.003709137,1,7.9,10,"San Jose, CA, US",0,
1633078062,"2021-10-01 08:47:42",-0.005778099,1,7.3,10,"San Jose, CA, US",0,
1633077314,"2021-10-01 08:35:14",-0.005929158,1,6.6,10,"San Jose, CA, US",0,
1633076557,"2021-10-01 08:22:37",-0.005826256,1,5.9,10,"San Jose, CA, US",0,
1633075814,"2021-10-01 08:10:14",0,-5,5.2,10,"San Jose, CA, US",,"i/o timeout"
1633074930,"2021-10-01 07:55:30",-0.005726415,1,10.7,10,"San Jose, CA, US",0,
1633074139,"2021-10-01 07:42:19",-0.005771249,1,10.3,10,"San Jose, CA, US",0,
1633073378,"2021-10-01 07:29:38",-0.005774511,1,9.7,10,"San Jose, CA, US",0,
1633072627,"2021-10-01 07:17:07",-0.005808687,1,9.2,10,"San Jose, CA, US",0,
1633071872,"2021-10-01 07:04:32",-0.004161128,1,8.6,10,"San Jose, CA, US",0,

It is possible to see that the problem has started just after 2021-10-01 08:47:42.

I guess there are more unreachable servers than used to be. The full monitoring round does not finish in quarter an hour.

Is there anybody else than @ask, who could verify what is going on with the monitor in San Jose?

2

Strange indeed. It seems to happen only on my IPv6 servers.

On https://web.beta.grundclock.com/ the frequency has not decreased in any way.

3

I can confirm the IPv6 behavior change:

image
image1182Ă—477 24.1 KB

This is based on all NTP requests sent by monsjc.
The corresponding IPv4 plot shows no such change.
The horizontal line at y= ~4000 seconds corresponds to servers having very poor scores. [the few that I checked had scores=-100
Many times the interval between poll is ~2 seconds: the major poll loop is roughly
Poll
<2 sec>
Poll
<2 sec>
Poll
<~900 sec>
I am not familiar with the actual implementation.
I do not know what happened on October 1.

1 Like
4

I see the same scenario in the monitoring history of my IPv6 server.

image
image724Ă—426 40.8 KB 

ts_epoch,ts,offset,step,score,monitor_id,monitor_name,leap,error
1633309646,"2021-10-04 01:07:26",-0.000895547,1,20,10,"San Jose, CA, US",0,
1633309646,"2021-10-04 01:07:26",-0.000895547,1,20,,,0,
1633304961,"2021-10-03 23:49:21",-0.000509637,1,20,10,"San Jose, CA, US",0,
1633304961,"2021-10-03 23:49:21",-0.000509637,1,20,,,0,
1633301352,"2021-10-03 22:49:12",-7.8077e-05,1,20,10,"San Jose, CA, US",0,
1633301352,"2021-10-03 22:49:12",-7.8077e-05,1,20,,,0,
1633298111,"2021-10-03 21:55:11",-0.000306721,1,20,10,"San Jose, CA, US",0,
1633298111,"2021-10-03 21:55:11",-0.000306721,1,20,,,0,
1633294385,"2021-10-03 20:53:05",-0.002362758,1,20,10,"San Jose, CA, US",0,
1633294385,"2021-10-03 20:53:05",-0.002362758,1,20,,,0,
1633286932,"2021-10-03 18:48:52",-0.000421467,1,20,10,"San Jose, CA, US",0,
1633286932,"2021-10-03 18:48:52",-0.000421467,1,20,,,0,
1633274943,"2021-10-03 15:29:03",-0.000572037,1,20,10,"San Jose, CA, US",0,
1633274943,"2021-10-03 15:29:03",-0.000572037,1,20,,,0,
1633272010,"2021-10-03 14:40:10",-0.000716877,1,20,10,"San Jose, CA, US",0,
1633272010,"2021-10-03 14:40:10",-0.000716877,1,20,,,0,
1633267742,"2021-10-03 13:29:02",-0.000809172,1,20,10,"San Jose, CA, US",0,
1633267742,"2021-10-03 13:29:02",-0.000809172,1,20,,,0,
1633263419,"2021-10-03 12:16:59",-0.000439117,1,20,10,"San Jose, CA, US",0,
1633263419,"2021-10-03 12:16:59",-0.000439117,1,20,,,0,
1633260012,"2021-10-03 11:20:12",-0.000399948,1,20,10,"San Jose, CA, US",0,
1633260012,"2021-10-03 11:20:12",-0.000399948,1,20,,,0,
1633256223,"2021-10-03 10:17:03",-0.002228708,1,20,10,"San Jose, CA, US",0,
1633256223,"2021-10-03 10:17:03",-0.002228708,1,20,,,0,
1633254281,"2021-10-03 09:44:41",-0.000340653,1,20,10,"San Jose, CA, US",0,
1633254281,"2021-10-03 09:44:41",-0.000340653,1,20,,,0,
1633248711,"2021-10-03 08:11:51",-0.000526385,1,20,10,"San Jose, CA, US",0,
1633248711,"2021-10-03 08:11:51",-0.000526385,1,20,,,0,
1633246429,"2021-10-03 07:33:49",-0.000169493,1,20,10,"San Jose, CA, US",0,
1633246429,"2021-10-03 07:33:49",-0.000169493,1,20,,,0,
1633241574,"2021-10-03 06:12:54",6.096e-05,1,20,10,"San Jose, CA, US",0,
1633241574,"2021-10-03 06:12:54",6.096e-05,1,20,,,0,
1633237971,"2021-10-03 05:12:51",2.9024e-05,1,20,10,"San Jose, CA, US",0,
1633237971,"2021-10-03 05:12:51",2.9024e-05,1,20,,,0,
1633235136,"2021-10-03 04:25:36",-0.001892125,1,20,10,"San Jose, CA, US",0,
1633235136,"2021-10-03 04:25:36",-0.001892125,1,20,,,0,
1633229739,"2021-10-03 02:55:39",-0.002051942,1,20,10,"San Jose, CA, US",0,
1633229739,"2021-10-03 02:55:39",-0.002051942,1,20,,,0,
1633226747,"2021-10-03 02:05:47",-0.000582162,1,20,10,"San Jose, CA, US",0,
1633226747,"2021-10-03 02:05:47",-0.000582162,1,20,,,0,
1633223719,"2021-10-03 01:15:19",-0.000876786,1,20,10,"San Jose, CA, US",0,
1633223719,"2021-10-03 01:15:19",-0.000876786,1,20,,,0,
1633219424,"2021-10-03 00:03:44",-0.000569935,1,20,10,"San Jose, CA, US",0,
1633219424,"2021-10-03 00:03:44",-0.000569935,1,20,,,0,
1633216430,"2021-10-02 23:13:50",-0.000799488,1,20,10,"San Jose, CA, US",0,
1633216430,"2021-10-02 23:13:50",-0.000799488,1,20,,,0,
1633214848,"2021-10-02 22:47:28",-0.000638741,1,20,10,"San Jose, CA, US",0,
1633214848,"2021-10-02 22:47:28",-0.000638741,1,20,,,0,
1633212532,"2021-10-02 22:08:52",-0.000115536,1,20,10,"San Jose, CA, US",0,
1633212532,"2021-10-02 22:08:52",-0.000115536,1,20,,,0,
1633210461,"2021-10-02 21:34:21",-7.6327e-05,1,20,10,"San Jose, CA, US",0,
1633210461,"2021-10-02 21:34:21",-7.6327e-05,1,20,,,0,
1633204846,"2021-10-02 20:00:46",-0.000466971,1,20,10,"San Jose, CA, US",0,
1633204846,"2021-10-02 20:00:46",-0.000466971,1,20,,,0,
1633199345,"2021-10-02 18:29:05",-0.000484116,1,20,10,"San Jose, CA, US",0,
1633199345,"2021-10-02 18:29:05",-0.000484116,1,20,,,0,
1633190590,"2021-10-02 16:03:10",-0.000418189,1,20,10,"San Jose, CA, US",0,
1633190590,"2021-10-02 16:03:10",-0.000418189,1,20,,,0,
1633188694,"2021-10-02 15:31:34",-0.001195027,1,20,10,"San Jose, CA, US",0,
1633188694,"2021-10-02 15:31:34",-0.001195027,1,20,,,0,
1633185716,"2021-10-02 14:41:56",-0.00108394,1,20,10,"San Jose, CA, US",0,
1633185716,"2021-10-02 14:41:56",-0.00108394,1,20,,,0,
1633179476,"2021-10-02 12:57:56",-0.000363655,1,20,10,"San Jose, CA, US",0,
1633179476,"2021-10-02 12:57:56",-0.000363655,1,20,,,0,
1633176062,"2021-10-02 12:01:02",-0.000823873,1,20,10,"San Jose, CA, US",0,
1633176062,"2021-10-02 12:01:02",-0.000823873,1,20,,,0,
1633171638,"2021-10-02 10:47:18",-0.000890765,1,20,10,"San Jose, CA, US",0,
1633171638,"2021-10-02 10:47:18",-0.000890765,1,20,,,0,
1633167678,"2021-10-02 09:41:18",-0.00072581,1,20,10,"San Jose, CA, US",0,
1633167678,"2021-10-02 09:41:18",-0.00072581,1,20,,,0,
1633158577,"2021-10-02 07:09:37",-0.000555373,1,20,10,"San Jose, CA, US",0,
1633158577,"2021-10-02 07:09:37",-0.000555373,1,20,,,0,
1633155758,"2021-10-02 06:22:38",-0.000759323,1,20,10,"San Jose, CA, US",0,
1633155758,"2021-10-02 06:22:38",-0.000759323,1,20,,,0,
1633153598,"2021-10-02 05:46:38",-0.000871001,1,20,10,"San Jose, CA, US",0,
1633153598,"2021-10-02 05:46:38",-0.000871001,1,20,,,0,
1633148425,"2021-10-02 04:20:25",0.000830172,1,20,10,"San Jose, CA, US",0,
1633148425,"2021-10-02 04:20:25",0.000830172,1,20,,,0,
1633141101,"2021-10-02 02:18:21",-0.000708537,1,20,10,"San Jose, CA, US",0,
1633141101,"2021-10-02 02:18:21",-0.000708537,1,20,,,0,
1633138397,"2021-10-02 01:33:17",-0.000773645,1,20,10,"San Jose, CA, US",0,
1633138397,"2021-10-02 01:33:17",-0.000773645,1,20,,,0,
1633133879,"2021-10-02 00:17:59",-0.000335129,1,20,10,"San Jose, CA, US",0,
1633133879,"2021-10-02 00:17:59",-0.000335129,1,20,,,0,
1633128220,"2021-10-01 22:43:40",-0.000622683,1,20,10,"San Jose, CA, US",0,
1633128220,"2021-10-01 22:43:40",-0.000622683,1,20,,,0,
1633124971,"2021-10-01 21:49:31",-0.000591518,1,20,10,"San Jose, CA, US",0,
1633124971,"2021-10-01 21:49:31",-0.000591518,1,20,,,0,
1633122495,"2021-10-01 21:08:15",-0.000501382,1,20,10,"San Jose, CA, US",0,
1633122495,"2021-10-01 21:08:15",-0.000501382,1,20,,,0,
1633115889,"2021-10-01 19:18:09",-0.000792448,1,20,10,"San Jose, CA, US",0,
1633115889,"2021-10-01 19:18:09",-0.000792448,1,20,,,0,
1633112506,"2021-10-01 18:21:46",-0.000569365,1,20,10,"San Jose, CA, US",0,
1633112506,"2021-10-01 18:21:46",-0.000569365,1,20,,,0,
1633110812,"2021-10-01 17:53:32",-0.000279557,1,20,10,"San Jose, CA, US",0,
1633110812,"2021-10-01 17:53:32",-0.000279557,1,20,,,0,
1633105803,"2021-10-01 16:30:03",-0.000608309,1,20,10,"San Jose, CA, US",0,
1633105803,"2021-10-01 16:30:03",-0.000608309,1,20,,,0,
1633100888,"2021-10-01 15:08:08",-0.000810103,1,20,10,"San Jose, CA, US",0,
1633100888,"2021-10-01 15:08:08",-0.000810103,1,20,,,0,
1633094899,"2021-10-01 13:28:19",-0.000546989,1,20,10,"San Jose, CA, US",0,
1633094899,"2021-10-01 13:28:19",-0.000546989,1,20,,,0,
1633089710,"2021-10-01 12:01:50",0.000747631,1,20,10,"San Jose, CA, US",0,
1633089710,"2021-10-01 12:01:50",0.000747631,1,20,,,0,
1633086270,"2021-10-01 11:04:30",-0.000725414,1,20,10,"San Jose, CA, US",0,
1633086270,"2021-10-01 11:04:30",-0.000725414,1,20,,,0,
1633082552,"2021-10-01 10:02:32",-0.000699617,1,20,10,"San Jose, CA, US",0,
1633082552,"2021-10-01 10:02:32",-0.000699617,1,20,,,0,
1633078626,"2021-10-01 08:57:06",-0.000243384,1,20,10,"San Jose, CA, US",0,
1633078626,"2021-10-01 08:57:06",-0.000243384,1,20,,,0,
1633077829,"2021-10-01 08:43:49",-0.0003932,1,20,10,"San Jose, CA, US",0,
1633077829,"2021-10-01 08:43:49",-0.0003932,1,20,,,0,
1633077063,"2021-10-01 08:31:03",-0.002027821,1,20,10,"San Jose, CA, US",0,
1633077063,"2021-10-01 08:31:03",-0.002027821,1,20,,,0,
1633076277,"2021-10-01 08:17:57",-0.001970034,1,20,10,"San Jose, CA, US",0,
1633076277,"2021-10-01 08:17:57",-0.001970034,1,20,,,0,
1633075529,"2021-10-01 08:05:29",-0.00067678,1,20,10,"San Jose, CA, US",0,
1633075529,"2021-10-01 08:05:29",-0.00067678,1,20,,,0,
1633074686,"2021-10-01 07:51:26",-0.000241573,1,20,10,"San Jose, CA, US",0,
1633074686,"2021-10-01 07:51:26",-0.000241573,1,20,,,0,
1633073908,"2021-10-01 07:38:28",-0.00062702,1,20,10,"San Jose, CA, US",0,
1633073908,"2021-10-01 07:38:28",-0.00062702,1,20,,,0,
1633073136,"2021-10-01 07:25:36",-0.000591053,1,20,10,"San Jose, CA, US",0,
1633073136,"2021-10-01 07:25:36",-0.000591053,1,20,,,0,
1633072342,"2021-10-01 07:12:22",-0.000785136,1,20,10,"San Jose, CA, US",0,
1633072342,"2021-10-01 07:12:22",-0.000785136,1,20,,,0,
1633071564,"2021-10-01 06:59:24",-0.000626697,1,20,10,"San Jose, CA, US",0,
1633071564,"2021-10-01 06:59:24",-0.000626697,1,20,,,0,
1633070836,"2021-10-01 06:47:16",-0.000743858,1,20,10,"San Jose, CA, US",0,
1633070836,"2021-10-01 06:47:16",-0.000743858,1,20,,,0,
1633070095,"2021-10-01 06:34:55",-0.000716905,1,20,10,"San Jose, CA, US",0,
1633070095,"2021-10-01 06:34:55",-0.000716905,1,20,,,0,
1633069368,"2021-10-01 06:22:48",-0.000989978,1,20,10,"San Jose, CA, US",0,
1633069368,"2021-10-01 06:22:48",-0.000989978,1,20,,,0,
1633068640,"2021-10-01 06:10:40",-0.000668841,1,20,10,"San Jose, CA, US",0,
1633068640,"2021-10-01 06:10:40",-0.000668841,1,20,,,0,
1633067862,"2021-10-01 05:57:42",-0.000788251,1,20,10,"San Jose, CA, US",0,
1633067862,"2021-10-01 05:57:42",-0.000788251,1,20,,,0,
1633066984,"2021-10-01 05:43:04",0.000957095,1,20,10,"San Jose, CA, US",0,
1633066984,"2021-10-01 05:43:04",0.000957095,1,20,,,0,
1633066237,"2021-10-01 05:30:37",-0.000725534,1,20,10,"San Jose, CA, US",0,
1633066237,"2021-10-01 05:30:37",-0.000725534,1,20,,,0,
1633065348,"2021-10-01 05:15:48",-0.000293137,1,20,10,"San Jose, CA, US",0,
1633065348,"2021-10-01 05:15:48",-0.000293137,1,20,,,0,
1633064611,"2021-10-01 05:03:31",-0.00078423,1,20,10,"San Jose, CA, US",0,
1633064611,"2021-10-01 05:03:31",-0.00078423,1,20,,,0,
1633063852,"2021-10-01 04:50:52",-0.000629364,1,20,10,"San Jose, CA, US",0,
1633063852,"2021-10-01 04:50:52",-0.000629364,1,20,,,0,
1633062997,"2021-10-01 04:36:37",-0.00093025,1,20,10,"San Jose, CA, US",0,
1633062997,"2021-10-01 04:36:37",-0.00093025,1,20,,,0,
1633062244,"2021-10-01 04:24:04",0.001041912,1,20,10,"San Jose, CA, US",0,
1633062244,"2021-10-01 04:24:04",0.001041912,1,20,,,0,
1633061443,"2021-10-01 04:10:43",-0.000684112,1,20,10,"San Jose, CA, US",0,
1633061443,"2021-10-01 04:10:43",-0.000684112,1,20,,,0,
1633060677,"2021-10-01 03:57:57",0.001144121,1,20,10,"San Jose, CA, US",0,
1633060677,"2021-10-01 03:57:57",0.001144121,1,20,,,0,
1633059849,"2021-10-01 03:44:09",-0.002068056,1,20,10,"San Jose, CA, US",0,
1633059849,"2021-10-01 03:44:09",-0.002068056,1,20,,,0,
1633059095,"2021-10-01 03:31:35",0.000750805,1,20,10,"San Jose, CA, US",0,
1633059095,"2021-10-01 03:31:35",0.000750805,1,20,,,0,
1633058332,"2021-10-01 03:18:52",0.000858825,1,20,10,"San Jose, CA, US",0,
1633058332,"2021-10-01 03:18:52",0.000858825,1,20,,,0,
1633057595,"2021-10-01 03:06:35",-0.000963394,1,20,10,"San Jose, CA, US",0,
1633057595,"2021-10-01 03:06:35",-0.000963394,1,20,,,0,
1633056818,"2021-10-01 02:53:38",-0.000618917,1,20,10,"San Jose, CA, US",0,
1633056818,"2021-10-01 02:53:38",-0.000618917,1,20,,,0,
1633056037,"2021-10-01 02:40:37",-0.000729683,1,20,10,"San Jose, CA, US",0,
1633056037,"2021-10-01 02:40:37",-0.000729683,1,20,,,0,
1633055278,"2021-10-01 02:27:58",-0.00058538,1,20,10,"San Jose, CA, US",0,
1633055278,"2021-10-01 02:27:58",-0.00058538,1,20,,,0,
1633054452,"2021-10-01 02:14:12",-0.000894863,1,20,10,"San Jose, CA, US",0,
1633054452,"2021-10-01 02:14:12",-0.000894863,1,20,,,0,
1633053724,"2021-10-01 02:02:04",-0.000761886,1,20,10,"San Jose, CA, US",0,
1633053724,"2021-10-01 02:02:04",-0.000761886,1,20,,,0,
1633052974,"2021-10-01 01:49:34",-0.000703444,1,20,10,"San Jose, CA, US",0,
1633052974,"2021-10-01 01:49:34",-0.000703444,1,20,,,0,
1633052163,"2021-10-01 01:36:03",-0.000269919,1,20,10,"San Jose, CA, US",0,
1633052163,"2021-10-01 01:36:03",-0.000269919,1,20,,,0,
1633051322,"2021-10-01 01:22:02",-0.0005656,1,20,10,"San Jose, CA, US",0,
1633051322,"2021-10-01 01:22:02",-0.0005656,1,20,,,0,
1633050571,"2021-10-01 01:09:31",-0.0003368,1,20,10,"San Jose, CA, US",0,
1633050571,"2021-10-01 01:09:31",-0.0003368,1,20,,,0,
1633049773,"2021-10-01 00:56:13",-0.000371724,1,20,10,"San Jose, CA, US",0,
1633049773,"2021-10-01 00:56:13",-0.000371724,1,20,,,0,
1633049008,"2021-10-01 00:43:28",-0.000703785,1,20,10,"San Jose, CA, US",0,
1633049008,"2021-10-01 00:43:28",-0.000703785,1,20,,,0,
1633048207,"2021-10-01 00:30:07",-0.000565441,1,20,10,"San Jose, CA, US",0,
1633048207,"2021-10-01 00:30:07",-0.000565441,1,20,,,0,
1633047373,"2021-10-01 00:16:13",-0.000261898,1,20,10,"San Jose, CA, US",0,
1633047373,"2021-10-01 00:16:13",-0.000261898,1,20,,,0,
1633046623,"2021-10-01 00:03:43",-0.00026184,1,20,10,"San Jose, CA, US",0,
1633046623,"2021-10-01 00:03:43",-0.00026184,1,20,,,0,
1633045770,"2021-09-30 23:49:30",-0.000284806,1,20,10,"San Jose, CA, US",0,
1633045770,"2021-09-30 23:49:30",-0.000284806,1,20,,,0,
1633044992,"2021-09-30 23:36:32",-0.000197534,1,20,10,"San Jose, CA, US",0,
1633044992,"2021-09-30 23:36:32",-0.000197534,1,20,,,0,
1633044234,"2021-09-30 23:23:54",-0.000277324,1,20,10,"San Jose, CA, US",0,
1633044234,"2021-09-30 23:23:54",-0.000277324,1,20,,,0,
1633043407,"2021-09-30 23:10:07",-0.000469842,1,20,10,"San Jose, CA, US",0,
1633043407,"2021-09-30 23:10:07",-0.000469842,1,20,,,0,
5

It seems that the problem is fixed at 2021-10-04 06:51:44:

ts_epoch,ts,offset,step,score,monitor_id,monitor_name,leap,error
1633340330,"2021-10-04 09:38:50",-0.005762062,1,12.7,10,"San Jose, CA, US",0,
1633339594,"2021-10-04 09:26:34",-0.005653501,1,12.3,10,"San Jose, CA, US",0,
1633338801,"2021-10-04 09:13:21",-0.005745455,1,11.9,10,"San Jose, CA, US",0,
1633338054,"2021-10-04 09:00:54",-0.00563033,1,11.5,10,"San Jose, CA, US",0,
1633337295,"2021-10-04 08:48:15",-0.005623531,1,11.1,10,"San Jose, CA, US",0,
1633336530,"2021-10-04 08:35:30",-0.004198586,1,10.6,10,"San Jose, CA, US",0,
1633335733,"2021-10-04 08:22:13",-0.00575173,1,10.1,10,"San Jose, CA, US",0,
1633334942,"2021-10-04 08:09:02",-0.005686088,1,9.6,10,"San Jose, CA, US",0,
1633334146,"2021-10-04 07:55:46",-0.005711201,1,9,10,"San Jose, CA, US",0,
1633333416,"2021-10-04 07:43:36",-0.005709228,1,8.4,10,"San Jose, CA, US",0,
1633332682,"2021-10-04 07:31:22",-0.005815228,1,7.8,10,"San Jose, CA, US",0,
1633331844,"2021-10-04 07:17:24",-0.004137309,1,7.2,10,"San Jose, CA, US",0,
1633331063,"2021-10-04 07:04:23",0,-5,6.5,10,"San Jose, CA, US",,"i/o timeout"
1633330304,"2021-10-04 06:51:44",-0.004150988,1,12.1,10,"San Jose, CA, US",0,
1633325721,"2021-10-04 05:35:21",-0.005879252,1,11.7,10,"San Jose, CA, US",0,
1633320990,"2021-10-04 04:16:30",-0.005704692,1,11.3,10,"San Jose, CA, US",0,
1633319661,"2021-10-04 03:54:21",-0.0040538,1,10.8,10,"San Jose, CA, US",0,

I wonder what happened.

6

Confirmed - I see it too. I wonder wat happened also.

7

There are massive attacks going on, mostly target DNS-servers.
Most of them originating from the USA, they overload all peers and DNS-servers.
E.g. whatsappweb.com can’t even been resolved or reached.
Many other sites are unreachable.

https://www.ntppool.org/scores/77.109.90.72

As usual, 1 monitor isn’t enough to monitor all NTP servers.
Servers decrease without any reason.

My IPv6 server has the same monitor problem.

https://www.ntppool.org/scores/2a00:7b80:477:21::9dcb:d174

The cause is always the same, poor San Jose peerings.

8

What happened is that Facebook made a mistake that brought their services (Facebook/Whatsapp/Instagram), including most of their authoritative DNS, down. In fact, the entire AS11917, containing the authoritative DNS for FB and WhatsApp disappeared from the internet.

As a result, many resolvers tried to resolve facebook.com and whatsapp.com in vain, which resulted in many retries, making the DNS somewhat unstable here and there. But this was not a deliberate, massive attack. Just fall out of the problems at FB.

I agree. That is why I hope the beta with multiple vantage points will be put in production some day soon.

9

By the way, Facebook NTP servers were affected by today’s (Oct 4, 2021) problems.

image
image739Ă—522 37.1 KB

I too have been lobbying for multiple monitoring points.
One caution. A client could still resolve a zone name into a particular NTP pool server that is poorly reachable/unreachable from that client.

1 Like
10

And the poll frequency is down again…

Apparently, no harm done, at least thus far. Without further information, it might be a harbinger of something about to go wrong.

Given the reliance on the pool by many, many worldwide, though the DNS infrastructure is seemingly well sized and reliable, its driver, the monitoring infrastructure, seems rather brittle, be it from routing or from peering issues.

Perhaps the monitoring servers should be distributed along with the DNS servers and exchanged the polling results with each other using Byzantine tallying for fault tolerance.

2 Likes
11

Hi Marco,

I saw the news, people complaining about Facebook, WA etc being down.
My thoughts, most people are so dumb, incredible your life is over when a stupid system is down.

I removed my Facebook account after a few months, that was 10 years back, never ever looked at that stupid platform again.
I do not rely on platforms, in fact they are useless. Only brain-dead people use that.
Same happened before with Napster and Citizen-band-radio.

People are soooooo stupid these days…brain-dead :grin:
They are like sheep, one jumps in the water, they all jump…incredible.

In Dutch we call our species mensDOM, translated as: humanSTUPID :yawning_face:

12

Let’s just hope the pool stays up forever! :muscle:t3: :grin:
(or a least as long as there is a need for it)

1 Like