Actually this is a very interesting thought.
Surely adding AAAA’s to ‘pool.ntp.org’ seems a perfectly safe thing to do, because that name is often used in conjunction with the ‘pool’ directive or with clients like ‘ntpdate’, and all of those a perfectly able to recover from unresponsive IP addresses (both v4 and v6).