Servers do heavy lifting in heavy lifting environments, streaming music files is not one of them unless you're Tidal or Qobuz. You’re better off keeping the server side completely away from the audio components connecting to a good streamer DAC like some mentioned here. Good DAC’s today filter noise and distortion and reclock the stream from the internal embedded clock as long as you’re not feeding it a garbage signal, use a decent NAS or server on a good strong network there’s no need to go bonkers price wise with the server side.
your thinking is about 3 years behind the current bleeding edge.
which is perfectly fine if you are happy. if you are wanting the best possible digital music reproduction you have some work to do.