Did you use pf or ipfw on freebsd (firewall machine) ?
After they added me in the CN zone last week, i saw alot of fw states going up towords 500k almost up to 1milion steady.
Thanks to avij when i saw what he wrote about notrack on the fw rules, i looked into the pfSense that im using and i have turned “state type” to “none” on those rules now, and only seeing about 100k towords 300k filter states.
I’m new into this and i think i have to disable the fw/nat in pfSense if want all filter states go away?
I want to have nat enabled and if someone has done this before, and want to give me a right direction on how i can do it, i will be deeply appreciative for any info i can get. Anyway, the fw has it alot better with 100k+ states
I use ipfw (which is actually fw2) on the Firewall/ntp server Firewalling was the original function of the machine, but it was running at such a light load that I added the PPS GPS (Garmin 18x LVC) to it and made it a public stratum 1 ntp server as well, then volunteered if for the pool. Apart from since the Chinese problems, it is still fairly lightly loaded, but does at least do a bit more to justify it’s electrical supply now
Yes, it may be possible that some Chinese DNS servers don’t obey the specified TTL of 150 seconds, but that is not the reason why the CN zone is overloaded. DNS servers (Chinese and elsewhere) don’t care if the NTP server in the DNS response is reachable or not. There simply isn’t such a feedback loop. The reason why there are problems is that there are zillions of pool users in China, and relatively few servers. Therefore those few servers will get a large amount of traffic, and if they aren’t configured properly, they will end up dying under the load. Serving a large amount of traffic may also be costly, and that may drive away some NTP server admins. My US server which is also in the CN zone sends around 2.5TB of NTP responses each month.
When your score drops below 10, your IP address is removed from pool.ntp.org rotation. This will typically drop your traffic to about a half (depending on how long your server has been in the pool etc) within 15 minutes or so. Those clients using software (such as ntpd) that do a lookup only at startup will continue requesting time from your server until the software or server is restarted. That traffic will also die down eventually, but much more slowly because some servers might be up for years without rebooting.
Many NTP server admins are surprised by the amount of traffic in some zones. It may very well be possible that big chunks of that traffic are abusive in nature, ie. misconfigured systems that poll the time every 10 seconds and such. Fixing those would of course be nice, but until that happens, the pool’s NTP servers will need to answer those NTP requests. If it isn’t your server, it’s going to be someone else’s server.
Even slower servers and routers can handle the traffic if configured properly. I don’t know if it’s possible in your environment, but one option might be to configure your router as a dumb bridge (ie. don’t assign it a public IP address at all) and handle the routing on a more capable server. That is usually the way I configure my home routers, so I don’t need to worry about my blood pressure trying to configure the cheap routers to do what I really want.
edit: Also, if you don’t have sufficient upstream bandwidth, that may also cause problems. 5000 pps is roughly 5Mbit/s. I don’t know what kind of request rates you were seeing.
There is an enormous amount of totally “wild” clients arround. Generally I have seen a huge amount of traffic like that (I just copied the first several entries few minutes ago):
ntpdc> mo remote address port local address count m ver code avgint lstint =============================================================================== 188.8.131.52 8764 184.108.40.206 3825272 3 4 1f0 0 0 220.127.116.11 58085 18.104.22.168 20907303 3 4 1f0 0 0 22.214.171.124 10062 126.96.36.199 10270354 3 4 1f0 0 0 188.8.131.52 34729 184.108.40.206 11434538 3 4 1f0 0 1 220.127.116.11 4360 18.104.22.168 575889 3 4 1f0 7 1 22.214.171.124 54103 126.96.36.199 624369 3 4 1f0 5 1 188.8.131.52 39086 184.108.40.206 9392631 3 4 1f0 0 1 220.127.116.11 2090 18.104.22.168 3991797 3 4 1f0 0 1 22.214.171.124 21884 126.96.36.199 7505310 3 4 1f0 0 1 188.8.131.52 30168 184.108.40.206 934158 3 4 5f0 4 1 220.127.116.11 24602 18.104.22.168 480341 3 4 5f0 6 2 22.214.171.124 16389 126.96.36.199 339827 3 4 5f0 9 2 188.8.131.52 4799 184.108.40.206 373675 3 4 1f0 8 2 220.127.116.11 7009 18.104.22.168 4715482 3 4 1f0 0 2 22.214.171.124 14389 126.96.36.199 551624 3 4 5f0 6 2 188.8.131.52 3054 184.108.40.206 703111 3 4 1f0 4 2
I did make quite an extensive list in ipf when I was trying to battle it, excluding partially even whole subnets, however, I did not take care about it for some time, and there are now, e.g. several IPs with over 10 million requests average 0 seconds, 220.127.116.11 is now on 21 million avg 0 sec.
From relatively recently (second half of 2016 approx.) we (the Internet) have the huge new problem of IoT cameras and many routers from a series of producers having fixed initial root/admin passwords which have to be changed by the end-user. Which they do not, many are probably even not aware of this need. What happend is that there is new virus spread, which tries to log from IP to IP, one after another, searching for any connected equipment with the fixed root/admin passwords. (I see an enormous amount of this ‘telnet’/‘ssh’ trying on my servers, which also sometimes influences the NTP servers availability.) After logging into such a (mostly) internet camera or home router, the Virus can install itself and any other software it likes (because the basis of probably all of that equipment is BusyBox).
Though I have no proof, I have a strong feeling that the enormous, tens of millions of times in a row, amount of NTP requests comes primarily from that equipment (and there are millions of such unprotected IoTs arround!), as I never before last year saw something like that.
The China zone is full of such misbehaving equipment, i tracerouted some of the incoming requests.
Presently my China zone server copes reasonably well (http://www.pool.ntp.org/scores/18.104.22.168), but you can see regular drops at certain times. However, I would not say that I would have any connectivity problems, as the internal network is multiple 100Mb/s and the external connection is 2 * 2 Gbit/s (if not upgraded last several years) (Rudjer Boshkovich Institute, Zagreb, Croatia). This server is a SunFire X4100 (2 * dual AMD Opteron, 2 GHz), Solaris 10. [About what Avij said about “even slower servers” I posted some time ago about my Sun 3/60 (MC68020+68881, 20MHz, 24MiB RAM), SunOS 4.1.1 acting already for years as part of ntp pool, and my web server (http://grgur.irb.hr/) ]
A (preliminary) list of client IPs requesting time milions of times constantly
Netherlands Zone Ziggo.nl Daily Traffic Spike
I just started a thread with an ip filtering rules list with the IPs of absolutely persevering clients:
This offer is still valid. I’d love to see more traffic on this server.
My old ntp4 will die soon (hardware will be used for other things).
Ask, can you add my new ntp4 to the cn pool please?
For those of you that have problems when you are in the cn pool:
Simply set your netspeed lower until you dont get any problems.
Its that easy.
For example, if you have a 1000 Mbit server in the cn pool you will get
approx 12000 pps and it will use approx 10 Mbit/s. If your hardware cant
handle that - set lower netspeed limit.
Please add this server to the CN zone as well:
Please add these servers to the CN zone:
I have added some of my server which is located in China.
Should i post them to this thread?
is this a current issue anymore?
If yes, you can add my new servers to the China or Brazil zone (or whatever zone is too weak to support the load).
I think you would get a faster response if you sent an email to server-owner-help (at) ntppool.org.
Im pretty sure ask sees it.
Sure, but there are also other people besides Ask who can do this, and they can be reached easiest by email.
And yes, I do think the China zone would still benefit from new servers.
I don’t know if it’s still one of the Pool’s most desperate zones, but Brazil definitely needs more servers. Even a server set to 384 Kbps gets a pretty huge amount of traffic during the times when it’s in the DNS.