@kennyc and @andy2 and a few others have stated it best.
The 0’s and 1’s that constitute the bits are actually processed with voltage pulses that are susceptible to not only jitter, but also EMI and RFI. While the introduction of noise will typically not cause playback errors or stuttering, digital signals are incredibly delicate, and any introduced noise will detract in a number of ways:
- Timing, which detracts from the focus and snap as if you have a photo just ever so slightly out of focus to a point where you wouldn’t realize unless you compared it side by side to another photo that is sharper/taken with a better lens.
- Harmonics, where the noise detracts from richness, affects tonal balance or liquidity of musical flow.
- Spatial presentation, where noise affects the size of the sound stage and the expansiveness and delineation of performers or elements in the recording.
- Effortlessness, where noise makes sound more etched, brash, or fatiguing.
Addressing noise is a key principle in high end audio, not only in streamers/servers, but in every component (phono stages, preamps, cables, room treatments), etc. With digital, as the signal is notably low in voltage, it takes much less noise to impact at a greater ratio. It takes incredible engineering as a result, via the design of sophisticated linear power supplies, advanced clocking, and isolation of noise from different components through architecture and chassis design to be able to combat the noise and deliver at a fidelity that rises above the typical streaming endpoint.
The high cost of a premium streamer/server is also not warranted for many audiophiles as they may not have a system that is able to take advantage of the auditory benefit they may bring. If someone is not hearing the difference between streamers, it is likely that they either do not have a system optimized to deliver a certain level of clarity or are not listening critically enough to care.
If you have been wanting to upgrade your streamer but have been disappointed with the results, you should be looking to address weak links in your chain first.
The same logic applies to everything even further upstream from the streamer, including routers, switches, Ethernet cables, etc.