We use chrony instead of ntpd, so can’t really help you there. I found that most of the work went into tweaking iptables/netfilter rules in order to not overload the conntrack tables (and thus drop UDP packets).
Compute wise, the servers (Xeon 4116) are living a good life (load ~1).