Welcome to the forum, @Macintoshdan!
NTP seems reachable fine, at least something is responding. Most of the time anyway.
But the time is off. When the offset exceeds a certain threshold, the score increase steps get smaller, become zero at 250ms offset, and then dip below zero (negative values, i.e., score decrease), with a maximum of two points subtracted eventually (or four, if the offset exceeds 3 seconds).
The graph pretty much shows it already, but looking at the numbers for one monitor as example, it is clear that the offset is somewhat outside the comfort zone of the scoring algorithm.
ts_epoch,ts,offset,step,score,monitor_id,monitor_name,leap,error
1749929016,2025-06-14 19:23:36,-0.899312969,-2,-28.857841492,69,usiad1-3grrbhg,,
1749928149,2025-06-14 19:09:09,-0.880236955,-2,-28.271411896,69,usiad1-3grrbhg,,
1749927336,2025-06-14 18:55:36,-0.862823873,-2,-27.654119492,69,usiad1-3grrbhg,,
1749926584,2025-06-14 18:43:04,-0.844263023,-2,-27.004335403,69,usiad1-3grrbhg,,
Compare that with the values seen for the IPv6 side of the server.
ts_epoch,ts,offset,step,score,monitor_id,monitor_name,leap,error
1749929219,2025-06-14 19:26:59,0.000492325,1,19.999847412,68,usiad2-3grrbhg,,
1749928623,2025-06-14 19:17:03,-0.000179601,1,19.999839783,68,usiad2-3grrbhg,,
1749927949,2025-06-14 19:05:49,-0.000646865,1,19.999832153,68,usiad2-3grrbhg,,
1749927262,2025-06-14 18:54:22,0.000176273,1,19.999822617,68,usiad2-3grrbhg,,
Are you sure the IPv4 traffic really reaches the same server as the IPv6 traffic? At least Miroslav’s NTP dedup tool doesn’t seem to think so, so maybe something else is responding to IPv4 requests, e.g., perhaps the router itself?
pi@RPi:~ $ ./ntpdedup.py -i 4 -w 10 -v 108.35.166.21 2600:4040:a1df:6400:8720:68fd:64d8:6af7
Iteration 1: responded 2/2 servers
Iteration 2: responded 2/2 servers
Iteration 3: responded 2/2 servers
Iteration 4: responded 2/2 servers
Duplicates:
Statistics:
IPv4 servers 1
with IPv4 duplicates 0 (0.0%)
with IPv6 duplicates 0 (0.0%)
IPv6 servers 1
with IPv4 duplicates 0 (0.0%)
with IPv6 duplicates 0 (0.0%)
Unique servers 2
I don’t fully understand the setup you have, but when you write
that seems to hint at it not being your RPi that is responding as well.