TL;DR: Your instincts are right. Load balancing NTP doesn’t make sense, and can do more harm than good.
RFC8633 (NTP Best Current Practice) has a section on anycast, and the same considerations apply to load balancers.
If you have only one public IP address, expose just one stratum 2 server to the pool.