The following is my opinion but it is based on pretty extensive long term empirical observation.
I have told this tale many times on various forums but here it is again.
My reference system includes a pair of B&W Nautilus 802 speakers, a dCS Delius DAC and dCS Purcell upsampler that I use to upsample CD to DSD. The stereo amplifier is a very, very large Mark Levinson circa the late 1990s--I forget the model number. All of the cabling is high-end Transparent Audio and I am also using their reference digital cable and power conditioner. This system has evolved through decades of meticulous, strategic trade-ins and upgrades.
The last CD transport I used was a very, very heavy, well built Goldmund Mimesis. Again, I don't recall the exact model number. I'm not in front of my system right now--it might have been the Mim39. This was the type of unit with a top that you lift up like a record player and a clamp that fits over the CD.
I sort of lost interest in the hobby for a while and then finally broke down and bought an iPod for use while on the go. I suddenly found I was listening to a lot more music again because of the functionality and I got curious about "servers". So I did some research and came up with the SlimDevices (now owned by Logitech) Squeezebox which is basically a small device that can recieve music files from your computer wirelessly or via ethernet. It can be used either with its onboard DAC or it can hand off a digital signal to an outboard DAC. I only use the digital coax output and wireless ethernet (wifi) for input.
I was expecting this unit to be a fun little toy for $300 that I would play around with but not as a replacement for a bona fide audiophile transport. So I was absolutely astonished to find that the difference in sound quality between the Squeezebox and the Goldmund was simply negligible, provided I used the same Transparent Audio digital cable. The Goldmund might have been a little sweeter and more nuanced, but in other ways I actually preferred the Squeezebox.
I had to seriously strain to hear the difference and if there was a difference, it was utterly outweighed by the functionality of Squeezebox.
So my answer to your question is, yes, you can use a hard drive based server system and get really, really good sound. However, you will get the best, most cost effective results if you think of the "server" as a replacement for the CD transport, and not a full front end system. It is just a pass through to get the music (the files should be uncompressed .wav files) into your DAC.
Once you go to a drive based system, there is no going back, in my opinion. The ability to access your entire library instantly on demand brings the hobby into the 21st century. For me it is a how did I live without this type of addition.
The problem is that everything depends on the performance of your DAC. If you have a good DAC, you will get good sound from a server. This is not exactly the golden age of the stand-alone DAC, however. The new dCS Paganini system, for example, is very music server unfriendly because unlike the older and cheaper Elgar generation products, there is no way to upsample an external digital source.
So the only limit to how good a server system can sound is how good the DAC is and all the associated components.
I do not subscribe to the notion of the "audiophile" music server for $10k. If you are going to spend $10k, spend it on a DAC, or a player with an onboard DAC that can accept an external digital source. Period. If an audio hardware reviewer is going to review a $10k server, they better compare it to $10k DAC.
I recall Stereophile comparing a $10k Linn server system to an Ayre Disc player that I believe is more like $5,000. That is not very helpful. The DAC is the most important part of the equation, and the Squeezebox proves that the serving can be done on the cheap, so a $10k Linn server should be compared only to a $10k DAC or disc player, because that is basically what it is. There is no inherent sound quality disadvantage to a hard drive based system if you are using uncompressed files. In fact, there may be advantages (Boulder's new $25k player is based on buffering a minute or so of data into RAM and feeding the DAC from the RAM, not directly streaming off the disc. What does that tell you?) Servers do not deserve, and should not be given special treatment as far as reviews and sound quality are concerned. They are basically DACS and should be treated as such.
The only criteria that really matter for servers are the quality of the user interface and supporting software, and the quality of the digital output on the server unit. There is no unit on the market right now that I am aware of that emphasizes those two things. So there might be room for improvement in these types of devices but the Squeezebox seems to do the job.