@marco.davids , this does seem to be a common setup and common issue, for residential servers anyhow. I have a stratum 1 on a residential cable internet provider and have been trying to figure this out recently with just this same type of behavior.
I’m surprised this circumstance has no clear step-by-step to achieving better performance from residential servers. the limitation seems to be conflict between need for NAT on residential NTP server and need to turn off connection tracking to get better performance.
constraints are:
(a) gotta keep internet working b/c work depends on it, so no wacky experimental device that bonks sometimes
(b) only 1 public IP address from a residential ISP