# Our servers suffer massive down-score - and I don't have the slightest clue why **URL:** \ **Category:** Server operators\ **Created:** [April 30, 2018, 9:11pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770 "2018-04-30T21:11:19Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ChrisW](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@ChrisW](https://community.ntppool.org/u/ChrisW)\ **Post date:** [April 30, 2018, 9:11pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/1 "2018-04-30T21:11:19Z")
Hi we’ve been running a pair of NTP servers for the pool for several years now. One of them high load (2 Gb setting, ~6 Mbps average) the second lightly loaded (and basically supposed to be switched in should #1 fail) These boxes have been up 24/7 for several years now, in a fully redundant data center with pretty good connectivity, and with standard operational monitoring. So imagine my surprise when Ask’s robot emailed me this morning telling me that Server #1 has been removed from pool because of low score. First I assumed the box had crashed - but it hadn’t. It was up (\> 1000 days uptime right now), ntpd was up and serving. Synchronization was ok too - it synchronized to a DCF77 Stratum1 box ~200 km north, and had a GPS based Stratum1 ~200km southeast as candidate. Both those Stratum1s are reliable Meinberg boxes. And it was claimed to have negative -13.3 score. Checking on the other, it also was scored pretty badly, plus 11.something (and has since fallen to 4.8) Being completely out of ideas I added another external stratum1 source and restarted ntpd on both boxes. And while box 1 is now very slowly creeping back top the 0 line, box 2 has since fallen way below the acceptability threshhold… Another thing I see is that they are monitored from the US West Coast. Both of these boxes are located in Central Europe (Frankfurt, Germany) - could it be that we are seeing here US west coast connectivity problems, not those of my boxes? See for yourself: [http://www.pool.ntp.org/scores/195.50.171.101](http://www.pool.ntp.org/scores/195.50.171.101) [http://www.pool.ntp.org/scores/195.50.171.102](http://www.pool.ntp.org/scores/195.50.171.102) Any idea? What can I do? ---
**Author:** ![marki](https://avatars.discourse-cdn.com/v4/letter/m/3da27b/32.png) [@marki](https://community.ntppool.org/u/marki)\ **Post date:** [May 2, 2018, 7:40am UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/2 "2018-05-02T07:40:22Z")
Seems like temporary network issue. Right now your NTP server is available worldwide - [https://atlas.ripe.net/measurements/12444930/#!probes](https://atlas.ripe.net/measurements/12444930/#!probes) ---
**Author:** ![curbynet](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/curbynet/32/147_2.png) [@curbynet](https://community.ntppool.org/u/curbynet)\ **Post date:** [May 2, 2018, 2:44pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/3 "2018-05-02T14:44:11Z")
Perhaps it’s related to this earlier issue? > [@Monitoring station seems to hate my server all of a sudden](https://community.ntppool.org/t/monitoring-station-seems-to-hate-my-server-all-of-a-sudden/738): > > So, over the last two days one of our locations NTP server seems to be having issues keeping a connection with the monitoring server. The server has been up for just shy of a year now, and previously there were no issues, and nothing in the setup has changed. We have thousands of active connections, and the server (GPS - Strat 1) is keeping good time, but I just can’t seem to keep a constant connection to the monitoring station. [ntp-server-monitoring-graph] [http://www.pool.ntp.org/scores/173…](http://www.pool.ntp.org/scores/173.161.33.165/log?limit=75) ---
**Author:** ![AlisonW](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/alisonw/32/268_2.png) [@AlisonW](https://community.ntppool.org/u/AlisonW)\ **Post date:** [May 2, 2018, 5:19pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/4 "2018-05-02T17:19:13Z")
> [@ChrisW](#): > > could it be that we are seeing here US west coast connectivity problems Very much so, yes. My server\* is connected by IPv4 and (native) IPv6 so, obviously, holds exactly the same time yet the graph is regularly different. - UK, dedicated machine. ---
**Author:** ![ChrisW](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@ChrisW](https://community.ntppool.org/u/ChrisW)\ **Post date:** [May 4, 2018, 5:58pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/5 "2018-05-04T17:58:18Z")
It started again. Box fell down to 4.6 this afternoon and is now slowly climbing back (now at 8.7) Are the Californian monitoring station affected by weekend traffic overload? What is their IP? What is their connectivity? Carrier? AS? Anything? And why is there no monitoring from Europe? ---
**Author:** ![Hedberg](https://avatars.discourse-cdn.com/v4/letter/h/e95f7d/32.png) [@Hedberg](https://community.ntppool.org/u/Hedberg)\ **Post date:** [May 4, 2018, 7:11pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/6 "2018-05-04T19:11:17Z")
I have 3 servers on 2 different ISP’s here in Denmark and they are all at 20 and has been so for days, so it is not a general problem. ---
**Author:** ![ChrisW](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@ChrisW](https://community.ntppool.org/u/ChrisW)\ **Post date:** [May 7, 2018, 4:35pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/7 "2018-05-07T16:35:58Z")
I am sorry guys, but this issue is still going on and I’m not closer to any solution. Would somebody please answer my question? What are the IPs of the monitoring systems ? What is their connectivity? Carrier? I need this to have our peering people look into that. ---
**Author:** ![ChrisW](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@ChrisW](https://community.ntppool.org/u/ChrisW)\ **Post date:** [May 11, 2018, 9:37am UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/8 "2018-05-11T09:37:32Z")
Hello Hedberg, as your “answer” seems to have effectively smothered any further discussion here, I have to state that this is of course a non-sequitur. The internet doesn’t work that way. Just because one place/ISP in Denmark has good connectivity to the US west coast monitoring stations has barely any implication on the connectivity of others there - or in neighboring countries. And I sill need the IP addresses of the monitoring stations. ---
**Author:** ![mlichvar](https://avatars.discourse-cdn.com/v4/letter/m/e79b87/32.png) [@mlichvar](https://community.ntppool.org/u/mlichvar)\ **Post date:** [May 11, 2018, 12:07pm UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/9 "2018-05-11T12:07:52Z")
You can find the address of the LA monitoring station in this thread: [https://community.ntppool.org/t/problems-with-the-los-angeles-ipv4-monitoring-station](https://community.ntppool.org/t/problems-with-the-los-angeles-ipv4-monitoring-station) Other stations are running in the new beta pool: [https://web.beta.grundclock.com](https://web.beta.grundclock.com) ---
**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\ **Post date:** [May 16, 2018, 3:06am UTC](https://community.ntppool.org/t/our-servers-suffer-massive-down-score-and-i-dont-have-the-slightest-clue-why/770/10 "2018-05-16T03:06:41Z")
Also, network debugging information here: [https://dev.ntppool.org/monitoring/network-debugging/](https://dev.ntppool.org/monitoring/network-debugging/) And yes, the beta site has more monitors and more details in the monitoring logs.