I've tried passive pre-amps before, as well as running my Benchmark DAC with it's own volume control directly in to my amps. The results have always been the same. At first I'm captivated by sense of transparency and zero haze, grain, etc. It really does sound more like the "straight wire with gain."
But every time I end up noticing it's almost "too transparent." That is, sonic images seem to be almost see-through, lacking the palbability, density, body and dynamics I get with a good pre-amp. I also find tonality a bit more believable with a good pre-amp.
I certainly haven't tried every passive option out there, but it seems many other audiophiles have had similar experience trying to go passive.