A few thoughts based on my experience, which over the last 25 years has followed a convoluted path, from integrated amp, to no preamp with no volume attenuation, to passive with buffer, to another passive with buffer, to endler resistor based volume control, to Promethius Ref 4 TVC, to Promethius Signature TVC, and finally to Coincident Statement Line Stage.
Based on my experience, one can certainly go without a preamp if one is single sourced. If the source has a digital volume control, that is good. No ability to attenuate the volume gets old pretty quickly. My path was driven by economics more than anything else. I think you can do very well in a carefully built system pretty cheaply using passive volume control, particularly transformer volume control designs. I would still prefer an excellent passive to an average active.
I don't think you can get to the ultimate level without a top notch active preamp. It is more than just volume control and source switching. You need the active pre to drive the amps. Most sources just can't do that well. There could be other issues at work, but the results in my system were undeniable. Everything--- dynamics, image, resolution, you name it. Every aspect of musical reproduction was improved by adding the Coincident CSL to my system.
It is my opinion that a top notch active preamp should be the ultimate goal in every system, money permitting. But one does have some pretty good relatively inexpensive options while on that path.