Unless Happy Eyeballs is implemented properly this is indeed the case. What will happen is: try IPv6 first, than try IPv4. This works without causing big problems. For instance, ‘ntpdate’ does this, so do ‘debian-ntpdate’ and ‘systemd-timesyncd’.
So does Mac OSX (not able to test Windows, since I have no access to such a device).
But ‘ntpdig’ (with and without the “-c” option) didn’t do the trick for me. It also has the “-4” option though, by which a problem can be solved. But ‘sntp’ on the other hand does a good job (I tried 1:4.2.8p12+dfsg-3ubuntu4.20.04.1).
The ntp-daemons (both NTPd en NTPsec) tend to stick to a (non-functional) IP address (v4 or v6, doesn’t matter) when the ‘server’ option is given, but when the ‘pool’ option is used instead, NTPsec is perfectly able to switch from (broken) IPv6 to (functional) IPv4 quickly. NTPd took a little longer, according to some preliminary testing I did. Chrony works well too with the pool directive, not so much with the server directive, so it seems. Same for openntpd; using the servers directive (with the ‘s’) recovers faster than ‘server’ (without the ‘s’).
I presume the behavior is the same as with IPv4.
The good news is; these ratios seem pretty good. From global, to regional, all the way to country-level; it’s actually pretty nice. Maybe some exceptions, like Brazil (25% IPv6 pool servers, compared to 40% IPv6 users), but that doesn’t seem to be a major problem. I already receive quite some IPv6-traffic on my IPv6-only servers in Europe from Brazil, so apparently this is covered by GeoDNS also.
Long story short; things are looking good. We could definitely add at least one more AAA to the pool (say 3.pool.ntp.org or so - probably 1.pool.ntp.org as well) and then gradually take it from there via region/country-zones, vendor-zones, 0.pool.ntp.org and finally pool.ntp.org itself.
Finally: let’s not forget; many major services are already reachable via IPv6 for a long time (Youtube, Google, Facebook and many others). Users will notice if their IPv6 is broken. So non-functional IPv6 connections, even though happy eyeballs tends to hide them from the user, are not nearly as common today as they where like a decade or so ago.
 If you want to test for yourself, here are three hostnames with broken (unreachable) IPv6-addresses: eyeball.forfun.net, eyeballs.forfun.net and happyeyeballs.forfun.net. Disclaimer: I may not keep them present in the zone forever, but for now I will leave them there for a while.