I’m wondering if it would be possible to have some sort of an API for changing server speeds? I have two use cases for this:
Automatically limiting used bandwidth to stay within monthly limits. My bandwidth quota from my ISP is around 6.5TB/month. Currently I have to keep an eye on the traffic graphs and adjust server speeds manually if I’m about to exceed (or stay too low of) my monthly limit. I used to have a neat way of doing this automatically, namely blocking NTP queries from the monitoring server when needed. The implementation of distributed monitoring broke this method. Case in point, I accidentally exceeded my monthly quota today and switched two servers of mine to “monitoring only” for the rest of the month to limit the extra costs.
One of my servers is fairly busy during the Chinese evening time (11-16 UTC), sometimes peaking at around 90k qps. During other times the query rate is a leisurely 20-40k qps. The problem is that when the query rates peak at 90k qps, the CPU usage hits 100%. I have not measured this, but I’d imagine running a NTP server at 100% CPU utilization might affect the quality of the NTP responses. I’d prefer to have the ability to adjust the speed setting dynamically so that the CPU usage could be something like 80% all the time, thus responding to around 72k qps around the clock.
I’m aware that this suggestion’s work/usefulness ratio might not be particularly good, but I thought I’d ask anyway.
Ouch! I know it’s not responsive to your question, but if it were me, I’d probably learn from an expensive lesson and stay farther away from using my whole quota. I agree API might be cool, and keep in mind the source code for the ntppool web site is on github, so you could right something up yourself that would work albeit at the risk of breaking when the website changes.
The speed you can set for your server isn’t an acutal speed.
It’s just the number of times it will be shown as surver when a DNS-request is done.
The speedsetting page does explain this:
Server settings
The net speed is used to balance the load between the pool servers. If your server can handle more or you’d like less NTP requests, you can adjust this value. The query load to your IP should adjust within 10-30 minutes. This speed does not mean the wire speed of your server, it’s just a relative value to other servers.
The amount of queries your server will get is generally directly proportional to the netspeed, so a 50Mbit setting will get about 5 times more traffic than the 10Mbit setting. Be aware that the amount of queries to your server will grow over time.
Hi Bas,
I don’t think he was confused on that point. He’s trying to automatically adjust his speed using a script to keep his usage under the amount his capped internet access provides. He can go over, but then he has to pay his ISP extra for the overage. So he’s looking for a web API endpoint to use, rather than simply hacking a script to emulate manually setting the speed using a browser. It’s a reasonable request, but given the demands on Ask’s time, I wouldn’t be surprised if other ntppool.org issues are higher priority for him.
But the problem is dat the ‘server-speed’ is just an estimate and only controls the DNS-round-robin of Ask.
Even if there is an API to change it, it will not help as there is no way to know for sure how much requests the DNS-system really gets.
Nameservers have stratum-caches like the NTP-pool has, the DNS of the NTP-pool has no control over that.
Ask can not control the DNS other then his OWN DNS-server, all the rest is caching and he has no way of knowing how much requests passes via those.
Ask makes an estimate based on his own DNS and alters the round-robin requests to sort of make it a guessed response on requests.
The real traffic is something you have to set your own, monitor it, set it lower if it’s too high and HOPE that it will be better.
Even at the lowest setting it can overload your server, there is no guarantee it will not happen.
Beware, if you set the setting lower, it will reduce the number of area’s that it will serve, he better look at that. If you serve the world, you can take a big hit when all get the same server.
As such I limit my servers to Europe + Country but not the World (@-sign).
I do not see how Ask can control this, not even with an API.
Yes, I’m indeed aware of the limitations of the “speed” setting. Let’s not stray too far away from the topic.
Right. The key point here is that if it looks like I’d exceed my monthly quota with the current transfer rate, I don’t want to manually open a web browser, log in to the pool management page and select the next lower setting from the pulldown menu for that NTP server.
Instead, I want to have a script that sends an authenticated query to my ISP’s API to retrieve the current network quota status and then figures out if changes to pool’s netspeed setting are needed. If so, the script would adjust the server’s speed setting on the pool’s management page by making a request to the hypothetical pool API. All this would be in crontab and would be executed maybe hourly or so, without me having to lift a finger to make it work past initial setup.
I already have the “querying ISP’s API for quota status” done (I needed it for my previous setup), but the latter part is what’s currently missing.
That’s not accurate. DNS responses include a time-to-live (TTL) in seconds that guides caching nameservers and resolvers on how long they can hold on to a cached DNS response before they need to flush it and repeat the query. pool.ntp.org responses put a 150s (2.5m) TTL on each server A or AAAA record. It’s easy to see this in action if you add “pool pool.ntp.org iburst” to your configuration file and restart ntpd/ntpsec. Assuming you don’t already have enough “server” entries to hit the default “tos maxclock 10” you’ll see 4 IPv4 servers from the pool spun up quickly, then 4 more added after at least 2.5 minutes, as a result of the first set of responses being flushed from DNS caches.
But I do not understand why you want to max out your bandwidth with a system that can not control this.
Just set it to where it stays below the limits you have.
Then leave it alone.
It’s not a competition to serve most people as possible.
I do not watch my traffic, I don’t, unless it hinders my own internet experience.
As I run an unlimited VDSL 30/100 business-account from home.
You can not control this, just set it about 50% lower, it will give plenty traffic.
I do not care about my traffic, not one bit…I just want to serve the system…and the system decides.
Why not just hack up code to emulate what the browser is doing when you manually adjust the speed setting? It should be relatively obvious from looking at the source of the management page where you can change the speed setting what you need to request or post to effect the speed change through the existing HTTP “API”.
Yes, but that is only the refresh a cache will get the new info. It doesn’t say when it does.
TTL means, after x time you need to get new data. With TTL 3600 that is 1 hour, but max 1 hour after your request. Worldwide this means 2 hours minus 1 second.
It will not change anything…the load isn’t regulated by the setting.
Why don’t you get this?
It’s an estimate where it controls the round-robin. IT DOES NOT REGULATE THE REQUESTS!
Ergo, if name-caches ignore the TTL, often they do, it won’t change anything and a big-hit can take a month or longer to last. TTL isn’t honored by everyone.
I probably should stop responding to you, as it seems it generates more heat than light. Changing the speed setting does change something – namely how frequently the server IP is included in DNS responses from the pool. But this is obvious to everyone, right? So what are you really trying to say? Perhaps that it’s only an indirect control on the traffic? Yes, I think everyone understands that.
The problem with queries continuing long after being removed from the pool is that ntpd and other NTP clients resolve once and keep querying. I have work in progress to improve that situation for ntpd, refining pool server selection based on which servers actually contribute to the time solution, and re-resolving server and pool names eventually so that (once the code is widely used) at least ntpd clients will stop using a pool server that’s removed from the pool eventually, regardless of whether the pool or server directive is used in ntp.conf.
As for writing some code for emulating what the browser does, the biggest hurdle is authentication. HTTP requests to change the server speed need to pass an “auth_token” value with the changed data. If there’s an easy way to obtain one I’m all ears.
I guess Protocols could be a starting point. Has someone already done something similar with the pool management page? I’d rather not reinvent the wheel.
The obvious way to me is to again emulate what the manual browser update process is – sign in with username and password to get that authentication token. It does require some reverse engineering, but all the source code you need should be available to you, either directly via the browser, or referring to:
The most surprising thing for me in this thread is that this approach worked. I’m curious which zones are you serving? More questions at the end.
I suspect the difference in experience with traffic shaping as a lot to do with which part of the world the server is in.
A traffic shaper for NTP is only useful if it’s possible to run out of quota with one setting of the control knob and stay under quota with a different setting; otherwise, the shaper is either on the maximum setting and under quota all the time, or over quota on the minimum setting all the time. At 1000 Mbps, a server in CA or US zones gets about 0.5% of the total pool traffic. The bandwidth setting covers more than 4 orders of magnitude, but the difference in traffic at the server is only about a factor of 5 between “1000 Mbps” and “monitoring only” (the variance over a day is larger than the effect of bandwidth settings). At the very highest setting, I can’t use even 10% of monthly quota on a single IP address. There have been schemes discussed in the forums that use multiple IP addresses at 1000 Mbps to get enough traffic to keep a single server busy. I went the other way: downsizing my NTP servers to match the load, or diverting any spare capacity to non-NTP services on the same hardware.
A traffic shaper has to be able to use whatever control it has early enough to avoid either bad outcome in spite of changes in traffic that it doesn’t control, and latency between the control setting and its effect on traffic. If I somehow used 80% of quota on NTP early in the month, I’d run out of quota at the end of the month, because “monitoring only” means roughly “reduce traffic to 20%”, and that traffic level persists for months after changing the setting. It’s usually possible to hit a quota target given a switch between “100%” and “20%” if the quota falls somewhere between those traffic levels. There can be no end-of-month burst of bandwidth to try to capture unused quota, since unpredicted increases in traffic level could persist into the following month(s) and cause those months to exceed quota too. The shaper would be always trying to avoid potential overage that only occasionally occurs (or the shaper overcommits, and the server owner pays overage charges occasionally).
Bas likely has a similar experience due to EU also having an abundance of servers or a shortage of clients.
In zones where there are too many clients and not enough servers (like China), there is a bigger share of clients per server, it’s possible (even likely) to exceed quota, and traffic shaping becomes not just relevant, but challenging: large variations in traffic, some of it at unpredictable times when only a few big servers in the pool go down…particularly if they’re all running traffic shapers that drop out of the pool near the end of a calendar month (mmmm…feedback loops!). In these zones, does a larger share of NTP traffic per server coincide with traffic levels being more responsive to the bandwidth setting? Do you have numbers that you can share on traffic changes as you change the bandwidth setting? What’s your strategy to avoid running out of quota in an end-of-month traffic surge? Does that strategy impact anyone else in the pool, and if so, how do you reduce that impact (e.g. save unused traffic to the end of the month, use the 10th or 20th as the cycle start/end date)?
@avij, you are pretty much spot on with recognizing that the biggest hurdle for this is managing authentication.
Each time I try to learn the Auth0 options for it, I don’t get super far. I think it’s more likely I’ll make some simple(r) OAuth token scheme. Anyway, you are right it doesn’t exist now and I’d want to see if I can get the zone generation across countries so the query load is more consistent. This is the first I’ve heard of the China query traffic being so variable through the day, that’s wild. We’ve lost a lot of servers from that zone, so that’s probably why your traffic has been going up: pool.ntp.org: NTP Servers in China, cn.pool.ntp.org
