That suggests you still have some connection tracking for NTP, possibly created by the server responses. The server traffic should not increase the number of connections at all. With nftables it needs two notrack rules, one for incoming direction, one for the outgoing direction, see the example in this post: Joining the pool kills my Internet - #21 by mlichvar
Use the nft list ruleset command to verify the rules generated from the OpenWrt firewall config.