A Raspberry Pi is actually extremely fast!
One of my time servers is a 1987 SUN 3/60, SunOS 4.1.1 !!!
The Sun 3/60 has a 32-bit MC68020 main processor and a 80-bit MC68881 FP coprocessor, running at 20MHz (approx. 3 MIPS, 0.4 MFLOPS), with 24 Mb of RAM. Presently I have 7 disk partitions of 2 Gb each (the max. size the OS can address), that is 14 Gb. Originally it was shipped with a 80 Mb or 120 Mb hard disk (including the whole OS, man pages, X-server and Openview WM, SunView windowing environment and a C compiler; previous OS versions had also a Fortran and Pascal compiler included). On 120 Mb there was quite a lot of space for private data.
The time-server net speed is set to 10Mb/s, which is the actual speed of the SUN 3/60 ethernet, and running in mid-europe (HR, EU, Global), it manages on average with around 10% of processor load, with maximums of around 25-30%! The machine is also a web server (http://grgur.irb.hr/).
This particular machine is almost constantly on the net from the early days of Janet in UK (around 1988/1989), and in Croatia from 1991/1992! During the 1990-ies it was a training machine for tens and tens of students connecting over terminals or X-terminals to it. I included it into the ntp pool in 2009.
So I wouldn't say a Raspberry Pi is a humble machine, except for the size and cost (though the SUN 3/60 is just like a sligltly overgrown pizza-box, that was the nickname of the housing for them anyway).
Sorry for the off-topic expose, but it is really that running a time-server must not be a hardware problem!