Hi Marc,
so you are running DD-WRT (great choice), so may have found this article: Router Slowdown - DD-WRT Wiki
They show how to increase the internal tables in the kernel to allow more IP/Port mappings. Decreasing the timeout may also help to clear the table faster.
But as you found out ultimately you will run into another limit and this time seemed to be the CPU in the router because NAT doesn’t seem to be hardware accelerated. So now other processes like DNSmasq are slowing down 
Unfortunately sometimes you lose, sometimes the other team wins…
Regards, Gunnar