Well the testing of the VPS with Chrony was successful I turned on 512 Kbit as the rate for IPv6 traffic only. I will eventually turn off the direct connection to my HomeLab but the difference between direct and VPS based on the monitors at the NTP project is showing a 2 to 3 millisecond offset which is better that direct to the HomeLab where I think latency comes from AT&T.
I am going to continue to monitor and maybe set the rate the 1.5 Mbps in the future. I got the smallest droplet at DigitalOcean I could get $6/month.
The end configuration looks like this:
GPS ->
Pi1 and Pi3 running GPSD/NTPsec configured with trusted sanity sources ->
WireGuard VPN ->
VPS with Chrony