In my experience chronyd needs slightly less CPU that ntpd, but most of the CPU time is the kernel receiving and sending the packets. Switching to chronyd will not help much with that.
To reduce the usage more, it may be necessary to increase the limits for interrupt coalescing with ethtool -C in order to reduce the number of interrupts. Of course, this has a negative impact on accuracy of NTP timestamps (unless the NIC supports and chronyd is configured to use HW timestamping). For a pool server that may be acceptable. For start, you could try setting rx-usecs to 100 and rx-frames to 20.
Some NICs have adaptive interrupt coalescing and changing the values with ethtool may not do anything. These should be able to handle larger packet rates in default configuration. For example, a machine with an Intel i210 card I use for testing can handle up to about 250kpps.
I’m not sure how much this is useful in virtual machines.