Thanks @Bas and @MagicNTP for showing me stateless NAT, I did not know. I used step 1-5 from Using an NTP-server behind NAT, it works! After testing a lot. No not many routers/OS can handle it. Really not many
My router did not struggle without stateless NAT, but now it struggle even less 
