This speed is typically for large packets, e.g., file upload/download, video streaming, … NTP is many small packets, which when handled on the CPU (vs. with hardware-acceleration) creates a bottleneck.
E.g., if your uplink is based on some “tunnel” protocol (such as PPPoE) vs., e.g., native IP over Ethernet, or if the device implements NAT or IDS/IPS or a firewall, that would typically run on the CPU on many types of devices.
E.g., note the difference in throughput for routed traffic for large and small packets for an example device, even with L3HW offloaded routing. Just to illustrate the general issue, not making a statement about that particular device or vendor, they just happen to publish that kind of information, but this is a general issue that affects more or less all devices. And the (scant) publicly available documentation of your device also has hints that it is not much different in this respect.
IDS/IPS/firewall are disabled as you write, what about NAT? Even if it is not used/needed, it might still be active. And even just plain L3 forwarding between IP subnets might be an “issue”, see example above.
So what would be interesting is what NTP traffic throughput you get on your server, e.g., at the 512Kbit/s “netspeed” setting?
The device you use seems to be enterprise grade, so more performant and functionally hopefully also more capable than a consumer device. Can you measure the traffic throughput on the outside of the device on the uplink side (towards your ISP)? Then, obviously, one could see a potential discrepancy between what comes from your ISP, and how much of that gets through your device and to your NTP server (or gets dropped by the device itself).
What is the HW/SW (OS, NTP implementation) of your NTP server?
In some way, the server phasing in and out of the pool might to some extent be considered a cosmetic issue only (red dots in the graph, and lines going up and down obviously don’t look good).
Sure, when there is overload, not only will the monitors notice packet drops, but so will actual clients. But clients are more tolerant to this than the monitors are.
And the pool system kind of implements a control loop to manage the acceptable load on the server, even if it is rather coarse right now, on and off only. (I previously suggested smoothing the adding and removing of traffic to servers a bit for just that reason, though I understand that seems not so easy to implement, based on feedback by Ask towards the (current) end of the thread.)
At least in the Singapore zone, from patterns observed there, it is my impression that some servers are run with too high a “netspeed” setting, and thus keep phasing into and out of the pool continuously. But one can still get good time from them. Not pretty, but still good enough for many/most clients.