That UniFi Dream Machine SE is acting as a firewall, isn’t it? In a later post, you mentioned blocking all traffic except a few countries (US and Korea, I’m guessing), that implies a firewall.
Take a look at the CPU and memory usage of the UDM. You can see it in the side panel under the insights tab for the router itself. Start at, e.g. https://[UDM_IP]/network/default/devices and click on the router, then on the side panel that pops up, the middle tab is insights. Expand the “System statistics” at the bottom to see it:
Turning off the geolocation features of the UDM should help with both CPU load and marginally the NTP delay.
You might be able to further decrease the router CPU load and the NTP delay by adding a firewall rule to allow all UDP destination port 123 in to all addresses and source port 123 out to all to short-circuit the firewall filtering.