No doubt, but this is about probabilities. When you drop or limit any kind of NTP traffic, it is always possible that someone's legit traffic will also get dropped. If you want to be 100% certain that all legit requests get answered, don't limit any traffic. However, that may also lead to your NTP server being used in a DDoS. So it's a compromise.
By your logic (port chosen randomly), packets from source ports 53 and 54 (which is apparently "xns-ch, XNS Clearinghouse") should occur approximately as often, right? On one of my servers, I received 402 packets from port 53 and 14 packets from port 54 in an hour. This gives me the impression that for many (but not all) of those packets originating from port 53, the port wasn't chosen randomly.
Edit: For full disclosure, the server received around 500k requests during that hour, so around 0.1% of requests got dropped due to that rule. I can live with this.