Since UDP is connectionless it’s easiest just to exclude inbound to 123 from being tracked (if possible).
Not just inbound - it’s important to make such a rule for the outbound packets also. Otherwise, the inbound traffic will be ignored for conntrack as per your original rule, and then the router will think that the outbound reply is actually a new connection and will start tracking when it sees that instead. The timeouts can be a bit different in this situation, because it looks like a non-replied connection rather than a reply-seen one (so it’ll usually drop out of the table faster), but it can still be something of a resource hog on the router if you’re seeing significant volumes of NTP traffic.
Also, avoid NAT if at all possible - most forms of NAT require connection tracking in order to work properly.