So if the streamer is always getting a perfect binary signal where does it go wrong?
Well the only thing I can think of is that noise is entering the system along with the binary stream. It's noise that's the polluter nothing else can be!
You're right it's noise of various kinds, e.g, but not limited to - RF, EMI, and high-impedance leakage currents from SMPS and network component clocks that increase jitter and clock phase noise.
The whole clocking thing is another space I haven't grasped enough to figure out how to make it right or even better
Thus there are some products available to address reducing the noise in our perfect binary signals
Here's a couple of links that describe the problem(s) and some products that improve the SQ
https://www.smallgreencomputer.com/pages/systemoptique-certified