Server score took a dive [oceania]

monitoring

#1

offset

11 servers are also taking a dive in the oceania region

$ traceroute 207.171.3.5
traceroute to 207.171.3.5 (207.171.3.5), 64 hops max, 40 byte packets
 1  * * *
 2  vl199-ds1-j2-r435.aus1.choopa.net (103.43.73.65)  16.122 ms  19.287 ms  20.384 ms
 3  * * *
 4  202.68.65.5 (202.68.65.5)  2.925 ms  13.438 ms  22.869 ms
 5  ae-8.r22.lsanca07.us.bb.gin.ntt.net (129.250.3.164)  173.549 ms  157.052 ms  157.707 ms
 6  ae-1.r01.lsanca07.us.bb.gin.ntt.net (129.250.3.123)  157.668 ms
    ae-1.r00.lsanca07.us.bb.gin.ntt.net (129.250.3.17)  157.925 ms  165.721 ms
 7  ae-1.a02.lsanca07.us.bb.gin.ntt.net (129.250.3.234)  161.658 ms  161.604 ms  163.332 ms
 8  te0-0-0-0.r04.lax02.as7012.net (198.172.90.74)  160.212 ms  160.842 ms  160.630 ms
 9  te7-4.r02.lax2.phyber.com (207.171.30.62)  160.202 ms  160.343 ms  160.289 ms
10  207-171-3-5.ntppool.net (207.171.3.5)  164.712 ms !Z  164.084 ms !Z  164.547 ms !Z

$ traceroute -P tcp  207.171.3.5
traceroute to 207.171.3.5 (207.171.3.5), 64 hops max, 40 byte packets
 1  * * *
 2  vl199-ds1-j2-r435.aus1.choopa.net (103.43.73.65)  3.673 ms  9.353 ms  10.995 ms
 3  * * *
 4  202.68.65.5 (202.68.65.5)  8.053 ms  0.513 ms  0.469 ms
 5  ae-8.r22.lsanca07.us.bb.gin.ntt.net (129.250.3.164)  157.893 ms  157.013 ms  149.801 ms
 6  ae-1.r00.lsanca07.us.bb.gin.ntt.net (129.250.3.17)  157.606 ms
    ae-1.r01.lsanca07.us.bb.gin.ntt.net (129.250.3.123)  157.372 ms
    ae-1.r00.lsanca07.us.bb.gin.ntt.net (129.250.3.17)  165.681 ms
 7  ae-0.a02.lsanca07.us.bb.gin.ntt.net (129.250.2.186)  164.804 ms  166.039 ms  157.715 ms
 8  te0-0-0-0.r04.lax02.as7012.net (198.172.90.74)  160.562 ms  152.711 ms  160.121 ms
 9  te7-4.r02.lax2.phyber.com (207.171.30.62)  159.736 ms  160.319 ms  159.576 ms
10  207-171-3-5.ntppool.net (207.171.3.5)  156.615 ms !Z  164.011 ms !Z  156.650 ms !Z

$ curl http://trace.ntppool.org/traceroute/45.76.113.31
Traceroute to 45.76.113.31
 1 ge1-10.hr02.lax2.phyber.com (207.171.3.1) AS7012  0.468  0.453  0.444
 2 te0-1-0-7.r04.lax02.as7012.net (207.171.30.61) AS7012  0.752  0.981  0.975
 3 hurricane-electric.as6939.any2ix.coresite.com (206.72.210.122)  0.487  0.493  0.485
 4 100ge8-2.core1.pao1.he.net (184.105.81.237) AS6939  8.307  8.311  8.469
 5 10ge4-1.core1.sjc1.he.net (72.52.92.114) AS6939  8.875  8.878  8.865
 6 vocus.gigabitethernet2-13.core1.sjc1.he.net (64.71.184.46) AS6939  10.232  10.215  11.445
 7 bundle-153.cor02.sjc01.ca.vocus.net (49.255.255.24) AS4826  159.239  159.516  159.296
 8 100g-0-0-0-0.cor01.sjc01.ca.vocus.net (49.255.255.4) AS4826  163.668  163.675  164.107
 0
 0
 0
12 xe-1-0-1.gw102.sy3.ap.equinix.com (27.111.240.147) AS17819  154.168  154.230
12 xe-1-0-1.gw102.sy3.ap.equinix.com (180.189.25.6) AS17819  152.428
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *

From man traceroute: !Z (communication with destination host administratively prohibited)


#2

To me it looks like somewhere the packets get lost or go to a network that has no idea where to send the packets (maybe due to a BGP configuration or update)

For comparison the beta system:


Notice how Zurich has a positive score

Update:
The server score is skyrocketing upwards.
It Looks like the issue was with equinix

$ curl http://trace.ntppool.org/traceroute/45.76.113.31
Traceroute to 45.76.113.31
 1 ge1-10.hr02.lax2.phyber.com (207.171.3.1) AS7012  0.642  0.617  0.595
 2 te0-1-0-7.r04.lax02.as7012.net (207.171.30.61) AS7012  0.925  0.920  0.907
 3 hurricane-electric.as6939.any2ix.coresite.com (206.72.210.122)  0.566  0.583  0.575
 4 100ge8-2.core1.pao1.he.net (184.105.81.237) AS6939  8.465  8.467  8.679
 5 10ge4-1.core1.sjc1.he.net (72.52.92.114) AS6939  8.678  8.985  8.959
 6 vocus.gigabitethernet2-13.core1.sjc1.he.net (64.71.184.46) AS6939  11.394  10.214  10.202
 7 bundle-153.cor02.sjc01.ca.vocus.net (49.255.255.24) AS4826  159.339  159.865  159.607
 8 BE-201.cor02.syd04.nsw.VOCUS.net.au (49.255.255.13) AS4826  164.051  159.391  159.616
 0
10 bundle-101.bdr01.syd03.nsw.vocus.net.au (114.31.192.87) AS4826  165.272  163.886  164.102
11 static-73.7.255.49.in-addr.VOCUS.net.au (49.255.7.73) AS4826  163.867  159.530  163.851
 0
 0
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *

#3

I’m seeing this on one of my public servers (http://www.pool.ntp.org/scores/103.35.80.142). There’s nothing wrong with the link, I can reach my server just fine from a VM in the UK, and another server on a different ISP is working just fine (http://www.pool.ntp.org/scores/150.101.178.79).

I’m going to put this down to a BGP issue between the monitoring system and my ISP.


#4

At this stage it’s a roller-coaster: offset

I’ve contacted Vultr and they confirm that my instance is reachable from various locations. @ask would you mind asking phyber is they can detect any problems on their end? interestingly enough the ipv6 address to the same instance is working without problems


#5

My servers are in .se and I have quite similiar issue, problems on IPv4 but not IPv6. I know that the servers
isnt overloaded so it seems to be some kind of BGP issue. My problems are not as serious as yours.
They started about the same time.

http://www.pool.ntp.org/user/iocc


#6

Mine seems to have recovered overnight.


#7

Actually, that’s not all that surprising that you are having no issues on IPv6. There’s actually no guarantee anywhere that IPv6 and IPv4 take the same route between two arbitrary nodes on the internet. Realistically, the first hop from your server to it’s configured gateway will be the same for both (though this is by convention, not because it’s required), but beyond that, it’s impossible to say (and depending on where the two nodes are located, it may be almost guaranteed that they take different routes).

I regularly see cases with my servers where only IPv4 or only IPv6 shows a few packets lost.


Increase in traffic, not just Australian servers