"i/o timeout" from different monitoring stations

Server operators
1

Hello,

Since a month or two, I’m getting very bad scores (due to i/o timeouts) on my NTP server (see https://www.ntppool.org/scores/81.83.23.34). I’ve also registered my NTP server to the beta pool, where it’s monitoring from different monitoring stations, but I also get many i/o timeouts (see https://web.beta.grundclock.com/scores/81.83.23.34). When I traceroute from and to the monitoring station, I get the following results:

sascha@kyle:~/monitor$ curl http://trace.ntppool.org/traceroute/81.83.23.34
Traceroute to 81.83.23.34
1 (139.178.64.41) AS54825 19.007 18.967
2 (147.75.98.104) AS54825 16.526
2 0.xe-0-0-17.dsr2.ewr1.packet.net (147.75.98.106) AS54825 72.081
3 (198.16.4.84) AS54825 0.383
3 0.ae21.bbr2.ewr1.packet.net (198.16.4.86) AS54825 0.744
4 0.et-0-0-0.bbr1.ewr2.packet.net (198.16.7.207) AS54825 0.810 0.793
5 (198.16.7.207) AS54825 1.274
5 nyk-b2-link.telia.net (62.115.175.182) AS54825 0.994
6 (62.115.118.115) AS1299 82.289
6 nyk-b2-link.telia.net (62.115.175.182) AS1299 1.180
7 be-zav01a-rb1-ae-21-0.aorta.net (213.46.162.6) AS6830 82.503 82.732
8 be-zav01a-rb1-ae-21-0.aorta.net (213.46.162.6) AS6830 * 82.666
9 * *
10 * *
11 * *
12 * *
13 * *
14 * *
15 * *
16 * *
17 * *
18 * *
19 * *
20 * *
21 * *
22 * *
23 * *
24 * *
25 * *
26 * *
27 * *
28 * *
29 * *
30 * *

sascha@kyle:~/monitor$ traceroute 139.178.64.42
traceroute to 139.178.64.42 (139.178.64.42), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 0.342 ms 0.228 ms 0.163 ms
2 * * *
3 * * *
4 dd5e0fa73.access.telenet.be (213.224.250.115) 9.420 ms 8.829 ms 9.266 ms
5 dd5e0fa24.access.telenet.be (213.224.250.36) 13.131 ms 13.015 ms 11.731 ms
6 be-zav01a-rb1-ae-19-0.aorta.net (213.46.162.1) 12.913 ms 9.127 ms 14.194 ms
7 213.46.162.5 (213.46.162.5) 14.322 ms 14.050 ms 13.901 ms
8 adm-bb3-link.telia.net (62.115.118.114) 95.103 ms adm-bb4-link.telia.net (62.115.118.118) 91.297 ms adm-bb3-link.telia.net (62.115.118.114) 94.957 ms
9 ldn-bb3-link.telia.net (213.155.136.98) 99.586 ms ldn-bb3-link.telia.net (62.115.113.211) 100.233 ms 100.833 ms
10 * nyk-bb4-link.telia.net (62.115.113.20) 100.021 ms 99.932 ms
11 nyk-b2-link.telia.net (62.115.137.99) 99.888 ms 96.998 ms nyk-b2-link.telia.net (213.155.130.28) 93.010 ms
12 packethost-ic-345229-nyk-b2.c.telia.net (62.115.175.183) 163.113 ms 163.192 ms 162.980 ms
13 0.et-11-1-0.bbr2.ewr1.packet.net (198.16.7.206) 92.559 ms 93.536 ms 93.389 ms
14 0.ae21.dsr1.ewr1.packet.net (198.16.4.87) 106.817 ms 117.765 ms 117.682 ms
15 147.75.98.107 (147.75.98.107) 96.748 ms 119.175 ms 147.75.98.105 (147.75.98.105) 96.846 ms
16 monewr1.ntppool.net (139.178.64.42) 95.812 ms 99.941 ms 99.974 ms

What can I do about this?

Thanks in advance for your answer.

2

Can you run a mtr to see if there is any packet loss?

3

Thank you for your answer. I’ve executed two mtr commands, which results in the following output:

sascha@kyle:~/monitor$ mtr -c 100 139.178.64.42 -rw
Start: 2019-10-07T14:46:28+0200
HOST: kyle.savanger.be Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.1.1 0.0% 100 0.3 0.3 0.2 0.4 0.0
2.|-- d51530001.static.telenet.be 0.0% 100 8.0 11.2 3.0 68.5 11.0
3.|-- dd5e0caf9.access.telenet.be 0.0% 100 8.9 8.2 4.2 23.6 2.5
4.|-- dd5e0fa73.access.telenet.be 0.0% 100 7.1 9.2 5.1 12.9 1.7
5.|-- dd5e0fa24.access.telenet.be 0.0% 100 8.3 12.5 7.8 23.4 2.4
6.|-- be-zav01a-rb1-ae-19-0.aorta.net 0.0% 100 15.0 12.3 7.8 31.8 2.7
7.|-- 213.46.162.5 0.0% 100 12.5 12.8 7.8 43.9 4.7
8.|-- adm-bb4-link.telia.net 0.0% 100 89.6 90.2 85.7 105.6 2.9
9.|-- ldn-bb4-link.telia.net 0.0% 100 96.4 94.7 89.5 109.0 2.8
10.|-- nyk-bb3-link.telia.net 1.0% 100 88.0 90.0 85.1 96.4 2.3
11.|-- nyk-b2-link.telia.net 0.0% 100 93.1 94.0 89.5 102.0 2.0
12.|-- packethost-ic-345229-nyk-b2.c.telia.net 0.0% 100 89.0 95.0 85.2 176.4 15.5
13.|-- 0.et-11-1-0.bbr2.ewr1.packet.net 0.0% 100 87.5 90.4 86.2 101.5 2.1
14.|-- 0.ae21.dsr1.ewr1.packet.net 0.0% 100 114.1 118.7 102.0 224.1 22.6
15.|-- 147.75.98.105 0.0% 100 109.0 114.5 103.2 203.4 15.0
16.|-- monewr1.ntppool.net 0.0% 100 91.5 94.7 90.3 105.5 2.1

sascha@kyle:~/monitor$ mtr -c 100 139.178.64.41 -rw
Start: 2019-10-07T14:50:32+0200
HOST: kyle.savanger.be Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.1.1 0.0% 100 0.4 0.3 0.3 0.4 0.0
2.|-- d51530001.static.telenet.be 0.0% 100 9.1 11.2 2.9 80.3 12.2
3.|-- dd5e0caf9.access.telenet.be 0.0% 100 8.9 8.2 4.0 15.4 2.0
4.|-- dd5e0fa73.access.telenet.be 0.0% 100 11.2 9.6 5.2 26.5 2.4
5.|-- dd5e0fa24.access.telenet.be 0.0% 100 10.9 12.1 7.6 17.5 1.8
6.|-- be-zav01a-rb1-ae-19-0.aorta.net 0.0% 100 13.8 12.6 7.6 27.9 2.5
7.|-- 213.46.162.5 0.0% 100 12.3 12.5 7.9 35.6 3.2
8.|-- adm-bb4-link.telia.net 0.0% 100 86.6 89.8 85.3 104.1 2.8
9.|-- ldn-bb4-link.telia.net 1.0% 100 94.8 94.5 90.0 108.9 2.8
10.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
11.|-- nyk-b2-link.telia.net 0.0% 100 94.2 94.2 89.3 120.4 3.6
12.|-- packethost-ic-345229-nyk-b2.c.telia.net 0.0% 100 89.7 96.0 85.2 180.1 18.7
13.|-- 0.et-11-1-0.bbr2.ewr1.packet.net 0.0% 100 91.2 90.1 86.2 98.2 2.0
14.|-- 0.ae21.dsr1.ewr1.packet.net 0.0% 100 112.9 115.9 101.9 221.3 22.4
15.|-- 139.178.64.41 0.0% 100 105.6 111.7 95.8 157.3 8.3

4

Here another MTR to the monitoring station of NTPPool, with UDP port 123… Seems very bad at all.

sascha@kyle:~/monitor$ mtr --udp -P 123 -c 10 139.178.64.42 -rw
Start: 2019-10-07T23:17:31+0200
HOST: kyle.savanger.be Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.1.1 0.0% 10 0.3 0.3 0.3 0.4 0.0
2.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- dd5e0fa73.access.telenet.be 0.0% 10 9.4 9.9 5.8 12.1 2.0
5.|-- dd5e0fa24.access.telenet.be 0.0% 10 12.2 12.2 8.2 14.8 2.3
6.|-- be-zav01a-rb1-ae-19-0.aorta.net 0.0% 10 9.6 16.3 9.6 48.2 11.5
7.|-- 213.46.162.5 40.0% 10 22.8 20.1 8.8 47.8 14.4
8.|-- adm-bb3-link.telia.net 50.0% 10 95.0 94.2 90.4 96.6 2.3
9.|-- ldn-bb3-link.telia.net 50.0% 10 98.6 97.3 94.1 100.2 2.4
10.|-- nyk-bb4-link.telia.net 70.0% 10 94.8 97.9 94.8 100.2 2.8
11.|-- nyk-b2-link.telia.net 60.0% 10 90.1 93.7 90.1 95.7 2.6
12.|-- packethost-ic-345229-nyk-b2.c.telia.net 40.0% 10 85.5 90.2 85.5 95.5 3.6
13.|-- 0.et-11-1-0.bbr2.ewr1.packet.net 60.0% 10 90.9 91.4 88.4 96.9 3.8
14.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0

5

Hmmm, looks like it starts with 213.46.162.5 which is Chello Broadband in BE? But Telia seems to also have issues on both sides on the atlantic…

Same hops too as the ones you posted above, so someone is actively dropping UDP packets and/or UDP/123 specifically… :frowning:

6

Thanks for your answer. Do you have any solution for this? Can I reroute the traffic via another way in my router (TP-Link Archer C7)? Do I maybe have to contact Chello Broadband/Telia? Or maybe @ask can investigate in this problem?

Thanks in advance for helping me out.

7

Received this mail… :anguished:

(This message is from the NTP Pool test system at https://www.ntppool.org/ )
Hi xxx,
Due to unresolved problems with your timeserver it is being removed from the NTP Pool (pool.ntp.org) project.
Server IP: 81.83.23.34 (kyle.savanger.be)
The server has been scheduled for removal on October 10, 2019.
If you perceive this to be in error (or fixed the server problem) and want to add the server to the project again, please go to pool.ntp.org: the internet cluster of ntp servers to use the automated system to re-add the server.
If you need help troubleshooting, the NTP Pool community at https://community.ntppool.org/ is very helpful, or you can reply to this email and our small team of volunteers will try to help.
If you haven’t logged in for a long time, you might need to recreate your account after our login system upgrade in 2016:
Login upgrade deployed | NTP Pool News
–
https://www.ntppool.org/

8

I’d suggest contacting your ISP and send them details of the mtr output to show where the traffic is trying to go and the problems you are seeing

9

Thanks for your answer. I’ve contacted my ISP, hopefully they have a solution for this annoying problem.

1 Like
10

Same here, it stops at packet.net

If I run the Curl command everything is fine, but mtr tells me it’s wrong:

mtr -u 139.178.64.42 -r

Then look at packet.net, the loss is near there or I eat my shoes :slight_smile:

I contacted packet-dot-net and they contact NTP to solve the problem.

1 Like
11

Enter your server in the beta-ntppool, see what happens.

http://web.beta.grundclock.com/

My “bad” server with minus scores almost got 20 at the beta server.
This means the monitor can’t be trusted, if you happen to follow a path that drops UDP-packages then your blue-line goes bad.

Nothing you can do except enter your server in the beta-pool and see if it’s better there, as that monitor is in Amsterdam and thus may follow a different path.

Mine is good in Amsterdam but bad in Newark.

See the difference in monitors, so it’s not on my side.

Normal-server
Normal-server.png758×681 59.3 KB

Then the Beta-server, very different, but also not good.

beta
beta.png632×600 37.6 KB

Greetings Bas.

12

Hi Bas, thanks for your answer. My servers are listed in the beta-pool (see the CSV-log @ https://web.beta.grundclock.com/scores/81.83.23.34/log?limit=200&monitor=*).
Unfortunately, the other monitoring servers at the beta-pool have also many timeouts to my NTP-server…

13

Same here, except it’s only Newark that is giving problems, Amsterdam and LA seem to be fine.

https://web.beta.grundclock.com/scores/77.109.90.72/log?limit=200&monitor=*

Yet my score it minus again, just because Newark has time-outs.

However, I did a trace on your UDP port 123 and it seems to be closed.

Did you open port 123 for UDP via My Telenet?

Greetings Bas.

14

Hi Bas, thanks for your answer. I have a modem-only from Telenet, so I have forwarded the port (both UTP & TCP) in my router and in the iptables-firewall on the NTP-Server. If I do different UDP port scans online, I can see that port 123 UDP is open. See https://check-host.net/check-report/b3a7175k2a7 for details. Also see https://suip.biz/?act=report&id=4828b8885be3bf9b49b0a3711019e168 for an NMAP-scan from suip.biz.

15

You only need to open UDP, not TCP.
The NTP server and it’s monitors only use UDP.
See how it goes.

16

Edited in my router forwarding config & in iptables @ my NTP-Server, still the same problem though…

17

Weird is it stops quickly after EDPnet…

bas@workstation:~/Bureaublad$ mtr -u 81.83.23.34 -p 123 -rw
Start: 2019-10-15T20:27:29+0200
HOST: workstation Loss% Snt Last Avg Best Wrst StDev
1.|-- wpad.fritz.box 0.0% 10 0.4 0.4 0.4 0.7 0.1
2.|-- bras01.bxl.be.edpnet.net 0.0% 10 7.9 8.1 7.9 8.4 0.2
3.|-- router01.bruix.be.edpnet.net 0.0% 10 12.7 9.9 8.5 13.3 1.7
4.|-- router02.bruix.be.edpnet.net 0.0% 10 8.5 8.9 8.5 9.7 0.4
5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
mtr: udp socket connect failed: Invalid argument
bas@workstation:~/Bureaublad$

Maybe Telenet is dropping the port?

18

Without giving a port, the same:

bas@workstation:~/Bureaublad$ mtr -u 81.83.23.34 -rw
Start: 2019-10-15T20:29:32+0200
HOST: workstation Loss% Snt Last Avg Best Wrst StDev
1.|-- wpad.fritz.box 0.0% 10 0.4 0.4 0.3 0.5 0.0
2.|-- bras01.bxl.be.edpnet.net 0.0% 10 7.9 8.1 7.5 8.4 0.3
3.|-- router01.bruix.be.edpnet.net 0.0% 10 8.7 8.9 8.5 9.5 0.3
4.|-- router02.bruix.be.edpnet.net 0.0% 10 8.8 9.0 8.5 10.1 0.5
5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
bas@workstation:~/Bureaublad$

19

UDP is a bit of a problem as the protocol has no return check.
As such it’s hard to check if it’s there or not.

20

I’ve emailed Telenet Business a couple of days ago, I hope they will/can solve the problem… Thankyou for helping me and I will keep you updated.