The incident rate of TCP 123 to other ports is about 5000 to 1, so portscans are most likely not the root cause for packets to TCP 123, even if they are a minor part the TCP 123 packets (about 1/5000)
I investigated this a bit more and found that quite some of them are in fact ntpv4 client requests which are just not UDP but TCP for some unknown reason.
It looks like some router/firewall is changing the UDP packet into a TCP one, while some of them are sending a TCP syn of 0 length, others send a TCP with full ntpv4 data body.
The 0 length TCP syn packets get re-sent usually 2 or 3 times about 1 second apart from each other.
I wonder what will come out of it if I answer them syn-ack, instead of just dropping them…