Should I eliminate my preamp?

"I can give you the maths if you want."

An offer that is hard to refuse, if not too complicated for my pay grade...

Here is some additional math:

With a 10K pot in series with the output coupling cap, a timing constant is introduced.

The formula for a timing constant is this:
1,000,000/Resistance x capacitance x 6.28 = F

Resistance is in ohms, capacitance in microfarads, F is frequency in Hertz and 6.28 is Pi x 2.

If you have a solid state source like a CD player, quite often it has a coupling cap at its output (probably an electrolytic capacitor). Depending on the designer's expectations, the cap is probably between 1uf and 10uf. But without knowing that, if we do know the amplifier's input impedance which has to be included, we have enough information to draw some conclusions.

If the amplifier input impedance is 10K (worst case scenario) we can see that the timing constant is raised an octave if the volume is all the way up. Depending on the design, this can introduce phase shift in the lower frequencies, which is perceived as reduced impact.

At volume control settings at less than full volume, a series resistance is introduced by the control. This effectively raises the output impedance of the source as far as the amplifier is concerned. To reduce the volume by 6 db, the control is at mid position, with 5K driving the output and 5K to ground. The series resistance thus introduced with such a low impedance load of 10K at the input of the amplifier can be audible as a reduction of dynamic punch at all frequencies.

Now the reason I mentioned the formula is that with different controls, the timing constants are different. Sometimes it works out well and sometimes it does not. But in all cases the interconnect cable is uncontrolled by such a high output impedance so its artifacts will be much more audible than if driven by a lower source impedance.

So audition is important- and its important to remember that while it might sound alright with a passive, it could well sound better (less colored for the reasons stated above) with an active. If you spent some money on the amps and speakers, it seems a shame to me to not get the most out of them.

This is an example of you get what you pay for. 

Yes, but the OP asked about using a DAC with volume control as a low impedance source to drive the interconnect to the amplifier. There is no passive attenuator in this scenario, and from an engineering standpoint no need for the preamp. The people arguing for retaining a preamp are arguing that they prefer the sound of the preamp. That may well be, but it is not more electrically accurate. The preamp is providing a subtle distortion that sounds pleasing to them. 

If you have a solid state source like a CD player, quite often it has a coupling cap at its output

What Ralph mentioned about coupling caps above is not relevant these days, as nearly all solid state sources have been direct-coupled for years, only tubed sources may have this problem as they have a caps on the output.
But below tubed preamps have this problem at the other end of the system with some of today's low input impedance amps.  

All is not good a match with 99% of tube preamps with  low input impedance amps these days, many Class-D's, some First Watt's, some Pass Lab's and many others.

The coupling cap that's needed on the output of 99% of tube preamps can be too small for these amps I mentioned, which can be as low as 10kohm input impedance. 

This is not a good match and will roll off the bass very early, to the point of owners saying this "tube pre has little or no bass".
  
To fix this one can change the cap for a much larger uF one which can be a fitting issue in itself or revert to a not so good sounding bi-polar electrolytic cap which will be small

Cheers George 

Willemj- how do you know what an album is supposed to sound like? Are you telling me by bypassing a preamp you are closer to the master? If you like vinyl these guys will tell you their master is more realistic. 
There are many reasons to go thru a preamp:
1) if your dac/cd player is solid state, you might want to go thru a tube preamp that front end solid state amps
2) you might want to upgrade to an external dac, add a tuner, turntable phono preamp, etc...

Bring home a couple of good preamps and compare them to your direct connect setup

You may think at first that you're hearing a more transparent signal but a preamp properly matches the impedance between stages and provides more muscle (better dynamics) in the long run if it's a fairly good pre which it sounds like you have. You will get more dynamics and you won't be as susceptible to roll-off of high frequencies. The pre-amp is more than just a handy switching device, it provides an easy load for whatever you plug into it and outputs a lower impedance which will make the amplifier happier than if you just plug in a source device.