# "io timeout" in CSV logs **URL:** \ **Category:** Server operators\ **Created:** [February 5, 2019, 12:37pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081 "2019-02-05T12:37:24Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 5, 2019, 12:37pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/1 "2019-02-05T12:37:24Z")
Hi, I want to ask about this “io timeout” that appears on csv log, i see the pontuation droping and connection dropping. What should be done at the server side to skip this errors? I’m operator of a stratum 1 NTP server that is a Raspberry PI 3 without original crystal oscillator (19.2MHz), we has connected the crystal input at one DDS that is connected to a local time scale UTC(LRTE) and is traceable in BIPM time department: [https://goo.gl/oSzRrF](https://goo.gl/oSzRrF) the server: [https://www.ntppool.org/scores/143.107.229.210](https://www.ntppool.org/scores/143.107.229.210) big drops was due server disconnection and settings changing because of upgrades, but, this “io timeout” always, always happened. Best regards, Luiz Paulo. ---
**Author:** ![alica](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/alica/32/337_2.png) [@alica](https://community.ntppool.org/u/alica)\ **Post date:** [February 5, 2019, 5:08pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/2 "2019-02-05T17:08:01Z")
Most probable reason is overwhelming inqueries rendering your server unresponsive. Try reduce your server bandwidth setting and see if it helps. ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 5, 2019, 7:45pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/3 "2019-02-05T19:45:19Z")
Thanks for reply! I changed it to 10MB/s, here we have a bandwidth of 1GB/s, but raspberry ethernet only supports 10/100 connection (Pi 3B not plus), did you think is good keep this value (10mb/s) or i can incrase it to something near to 100mb/s? ---
**Author:** ![kennethr](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/kennethr/32/89_2.png) [@kennethr](https://community.ntppool.org/u/kennethr)\ **Post date:** [February 5, 2019, 7:51pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/4 "2019-02-05T19:51:07Z")
The **net speed** is used to balance the load between the pool servers. If your connection is asymmetric (like most DSL connections) you should use the _lower_ speed. The amount of queries your server will get is generally directly proportional to the netspeed, so a 50Mbit setting will get about 5 times more traffic than the 10Mbit setting. The pool will only use a fraction of the “netspeed setting”. Be aware that the amount of queries to your server will grow over time. ---
**Author:** ![littlejason99](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/littlejason99/32/242_2.png) [@littlejason99](https://community.ntppool.org/u/littlejason99)\ **Post date:** [February 5, 2019, 8:48pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/5 "2019-02-05T20:48:23Z")
Depending on which NTP program you use, there are commands to monitor performance and you can see if it’s dropping packets or if you are just experiencing a network issue between you & the monitoring server. ---
**Author:** ![ask](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/ask/32/907_2.png) [@ask](https://community.ntppool.org/u/ask)\ **Post date:** [February 7, 2019, 4:06pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/6 "2019-02-07T16:06:04Z")
The program that does the monitoring is at [https://github.com/ntppool/monitor](https://github.com/ntppool/monitor) - review and patches are welcome. The production system is configured to just do “one probe”. The old system did more (sorta). I have been working on making the long term logs more useful (and accessible). In the process I noticed that the “worse monitoring” started around March last year (and not during the summer when various network changes were done, to my surprise). It used to be that about 0.5% of the monitoring probes to servers that have recently worked would fail. This is still the case for IPv6, but for IPv4 servers it’s gone up to around 1%. (I’ll most the queries and data in more detail later when I’m done with the data processing; the server doing it is using terabytes of (compressed) space temporarily, it’s pretty slow to work with). ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 5:18pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/7 "2019-02-09T17:18:54Z")
I’m using ntpd from raspbian. ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 5:23pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/8 "2019-02-09T17:23:59Z")
Seems more stable now, i’ve dropped server speed configuration to 768kbit, this is a little shameful but it keeps server available. Also, tunned some parameters at sysctl in server side. Like: net.netfilter.nf\_conntrack\_udp\_timeout = 120 net.netfilter.nf\_conntrack\_udp\_timeout\_stream = 720 net.netfilter.nf\_conntrack\_buckets = 48256 net.netfilter.nf\_conntrack\_max = 262144 net.nf\_conntrack\_max = 262144 net.core.netdev\_max\_backlog = 350000 net.core.rmem\_max = 38414400 net.core.rmem\_default = 38414400 net.ipv4.udp\_rmem\_min = 24384 net.ipv4.udp\_wmem\_min = 24384 net.core.default\_qdisc = fq net.ipv4.tcp\_congestion\_control = bbr net.core.somaxconn = 16384 i dont know if tcp\_congestion\_control changing makes a real effect at ntp, concerning this uses upt protocol. But, you guys have some another recommendation? The server is in one public university, that we have a real IP addres with full internet access, now i’m measuring a conntrack count of 220324 what is a huge value, stock is arround 65536… What did you think about? ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 5:30pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/9 "2019-02-09T17:30:49Z")
I saw here the internet distribution system, the connection is symmetric. ---
**Author:** ![kennethr](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/kennethr/32/89_2.png) [@kennethr](https://community.ntppool.org/u/kennethr)\ **Post date:** [February 9, 2019, 5:48pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/10 "2019-02-09T17:48:31Z")
You should not use conntrack on ntp Here is an example to use NOTRACK with iptables: where xxx = ip sudo iptables -t raw -A PREROUTING -d xxx.xxx.xxx.xxx -p udp --dport 123 -j NOTRACK sudo iptables -t raw -A OUTPUT -s xxx.xxx.xxx.xxx -p udp --sport 123 -j NOTRACK see if you can set an higher net speed after this. you can also count nf\_conntrack, try before and after adding to iptables with command: cat /proc/net/nf\_conntrack | wc -l ---
**Author:** ![kennethr](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/kennethr/32/89_2.png) [@kennethr](https://community.ntppool.org/u/kennethr)\ **Post date:** [February 9, 2019, 6:06pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/11 "2019-02-09T18:06:37Z")
If you are behind a NAT router, then you will also need to check if you can turn off conntrack on the router, or raise the state count. ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 6:07pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/12 "2019-02-09T18:07:48Z")
I’m not using a NAT router, its a direct ip address, 143.107.229.210 if you want to check it. About these commands, what they should do? ---
**Author:** ![kennethr](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/kennethr/32/89_2.png) [@kennethr](https://community.ntppool.org/u/kennethr)\ **Post date:** [February 9, 2019, 6:16pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/13 "2019-02-09T18:16:04Z")
It disables the conntrack, so it does not save states off the connections, since UDP is stateless, it does not require a state in the table ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 6:19pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/14 "2019-02-09T18:19:51Z")
can i still using conntrack -L -p udp | wc -l to show it? It drops from 200000 to 200 and incrasing. ---
**Author:** ![kennethr](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/kennethr/32/89_2.png) [@kennethr](https://community.ntppool.org/u/kennethr)\ **Post date:** [February 9, 2019, 6:29pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/15 "2019-02-09T18:29:49Z")
you can still use the command, but it will not count port 123, since it will exhaust the server with all the connections. so if you get around 200 - 1000, than it works, then it only show all other connection than ntp 🙂 ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 6:32pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/16 "2019-02-09T18:32:20Z")
Nice!! So, saving the udp connection states it decrases server performance? Why? ![Captura%20de%20tela%20de%202019-02-09%2016-25-20](https://us1.discourse-cdn.com/flex016/uploads/ntppool/original/1X/cc7dbdab412d64f443777d3ca660bfb04b0fc2ae.png) ![Captura%20de%20tela%20de%202019-02-09%2016-25-09](https://us1.discourse-cdn.com/flex016/uploads/ntppool/original/1X/0a2aa1b66a98879c8ce5ac4d7e3dc6cfb39a153f.png) ---
**Author:** ![kennethr](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/kennethr/32/89_2.png) [@kennethr](https://community.ntppool.org/u/kennethr)\ **Post date:** [February 9, 2019, 6:40pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/17 "2019-02-09T18:40:10Z")
It all depends on how much memory and cpu you have. An exampel is: Each state consumes approximately 1KB of RAM. So for 1,000,000 states, 1GB of RAM would be required just for states. Each _connection_ flowing _through_ the firewall will consume two states: One state when entering, and one state when exiting. So when high number of states it have to handle, it also can affect peformance. Thats why its a good thing to turn on NOTRACK on NTP ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 6:42pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/18 "2019-02-09T18:42:57Z")
very intersting, i will use this too for my openwrt rpi server in my home, as this rpi ntp server i only have 1gb of ram… And… in accordance with this logic, logging this can eat a lot of RAM!! ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 8:57pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/19 "2019-02-09T20:57:16Z")
Some pictures from my setup: Symmetricom 5071; HROG 10 Micro Phase Stepper; Stanford DDS; Raspberry pi 3 running without crystal (external 19.2MHz locked to our UTC(LRTE) from HROG) and PPS input. The monitoring software is from our GPS receiver to compare local frequency with GPS frequency and keep our time tracked to UTC. ![photo5114224791975471089](https://us1.discourse-cdn.com/flex016/uploads/ntppool/original/1X/cc75c5598b5d91ea780e273bbfce4023cab3aa93.jpeg) ![photo5114224791975471088](https://us1.discourse-cdn.com/flex016/uploads/ntppool/original/1X/561456cc8498842b6378334873649d343690068e.jpeg) ![photo5114224791975471087](https://us1.discourse-cdn.com/flex016/uploads/ntppool/original/1X/ac06f8f2273435e05811dc7ac7d156d600e9c420.jpeg) ![photo5114224791975471086](https://us1.discourse-cdn.com/flex016/uploads/ntppool/original/1X/0d576c2a1adc65d401e03663493a9d85be8d7cb2.jpeg) ![photo5114224791975471085](https://us1.discourse-cdn.com/flex016/uploads/ntppool/original/1X/3794b46866de35a9e48c5eb9a7a3d8c5220a8be2.jpeg) ---
**Author:** ![electronicslover](https://sea2.discourse-cdn.com/flex016/user_avatar/community.ntppool.org/electronicslover/32/283_2.png) [@electronicslover](https://community.ntppool.org/u/electronicslover)\ **Post date:** [February 9, 2019, 9:00pm UTC](https://community.ntppool.org/t/io-timeout-in-csv-logs/1081/20 "2019-02-09T21:00:22Z")
Maybe this can run in this Pi server? Or needs a powerful hardware?