If the 3 gpbs setting only resulted in 600 packets per second - it’s not too much bandwidth for your 100/35 mbps VDSL connection
The NTP packet isn’t larger than 100 bytes - I haven’t had DSL is over 25 years, but I think the frame size back then was 1492 bytes - MTU isn’t an issue
Sounds like IPfire is being overloaded with states and not clearing them fast enough
