Do Streamer only devices really impact sound quality?

If you are going to use usb to a dac (why anybody would is beyond me), then a server will probably make a difference. If you use Ethernet then IME they don’t and you don’t have any issues with jitter or clocking. It’s only with usb that you have to buy all the tweaks/gimmicks to try to get usb to sound good. 
You still have to provide a quality internal network infrastructure with top quality cables to get good quality sound using Ethernet.

For over a decade, the software that is used to read the data has more influence on sq than the hardware. Even when I used an external server years ago, the Auralic Aries, I preferred the sq when I used the Lumin software to access the Aries. Same goes for iTunes, pure music, amarra, audirvana, and Roon. 

Individuals use usb because it has best SQ vs other inputs/outputs available to them. I've observed many trying coax, AES/EBU, I2S inputs on dac, finding them inferior and returning to usb. Theoreticals are one thing, hearing is another. USB continues to be used because both manufacturers and consumers find it preferable to other formats.

As for software vs hardware, I've heard great variability with both.

My take is optimized streaming only achieved when all components, software, software and hardware configurations are optimized. Everything matters. Based on reports of others I observe great correlation between optimized streaming setups and great sound quality. The most experienced streamers report their findings, others follow their recommendations and experience better sound quality in vast majority of cases.

The DAC is much more important. I like to go optical off the streamer to prevent noise from being transmitted from the streamer. 

Individuals use usb because it has best SQ vs other inputs/outputs available to them. I've observed many trying coax, AES/EBU, I2S inputs on dac, finding them inferior and returning to usb

@sns IME no…just, no.  That may be your individual experience and can always be gear dependent, but other than Holo DACs I don’t know of any other DAC manufacturer that recommends USB over i2S or AES/EBU.  For example, I fed my Musician Pegasus DAC via a high-quality USB cable and then bought a Denafrips Iris that let me use the Pegasus’ i2S input and it was a whole new world of performance and I ain’t no way going back to USB.  What USB versus i2S or AES/EBU connections have you tried personally and with what equipment?  You can cite “other people’s” experience all you want, but unless you’ve done it yourself, as I have, you’re just propagating hearsay and nothing more.  Back up your, frankly, controversial contentions with actual experience or this is just audio flimflam based on nothing and counter to what most have experienced here. 

Didn't Alvin Lee state that when using the Node with any of the Denafrips streamers that USB was the preferred connection? I think I read that here on AG, but can't find the specific post or remember the reasoning for his comment .