It does that already, and monitors that seem to be off too much with respect to external references are ignored. But that should only be a safeguard for exceptional circumstances, e.g., networking or local system issues, but not compensate for a lack of basic good timekeeping performance.
Sure, there is no guarantee. But just because there are other sources of instability doesn’t mean that the implementation itself should unnecessarily be one, too.
The basic timekeeping setup should be as stable as possible, including the timekeeping daemon having typically known good performance, so that the rest of the system only has to deal with unavoidable external issues.