Better, just let NTP traffic bypass the connection tracker to eleminate those problems at the root 
iptables -t raw -I PREROUTING -p udp -m udp --dport 123 -j CT --notrack
iptables -t raw -I PREROUTING -p udp -m udp --sport 123 -j CT --notrack
iptables -t raw -I OUTPUT -p udp -m udp --dport 123 -j CT --notrack
iptables -t raw -I OUTPUT -p udp -m udp --sport 123 -j CT --notrack
This will simply let the connection tracker just ignore all traffic from or to port 123/UDP.
But, of course, you then cannot use “-m state” for those ports and you need to make sure that both outgoing and incoming traffic (answers) can flow through your firewall.