It depends on a lot of unstated details, so it’s not really possible to say simply yes or no.
For a pool server, accuracy in the single-digit milliseconds is more than enough to stay in the pool. For a pool monitor, on the other hand, I’d want appreciably less than 1 msec offsets seen locally, and as symmetric path as possible to handoff(s) to other ISPs (AS numbers).
There’s a previous thread here that discusses the age-old “bare metal only for NTP” which has become less mandatory depending on details like type of hypervisor, CPU virtualization support, etc: Cringe at thought of NTP server on a VM?