Should I eliminate my preamp?

Yes, but the OP asked about using a DAC with volume control as a low impedance source to drive the interconnect to the amplifier. There is no passive attenuator in this scenario, and from an engineering standpoint no need for the preamp. The people arguing for retaining a preamp are arguing that they prefer the sound of the preamp. That may well be, but it is not more electrically accurate. The preamp is providing a subtle distortion that sounds pleasing to them. 

If you have a solid state source like a CD player, quite often it has a coupling cap at its output

What Ralph mentioned about coupling caps above is not relevant these days, as nearly all solid state sources have been direct-coupled for years, only tubed sources may have this problem as they have a caps on the output.
But below tubed preamps have this problem at the other end of the system with some of today's low input impedance amps.  

All is not good a match with 99% of tube preamps with  low input impedance amps these days, many Class-D's, some First Watt's, some Pass Lab's and many others.

The coupling cap that's needed on the output of 99% of tube preamps can be too small for these amps I mentioned, which can be as low as 10kohm input impedance. 

This is not a good match and will roll off the bass very early, to the point of owners saying this "tube pre has little or no bass".
  
To fix this one can change the cap for a much larger uF one which can be a fitting issue in itself or revert to a not so good sounding bi-polar electrolytic cap which will be small

Cheers George 

Willemj- how do you know what an album is supposed to sound like? Are you telling me by bypassing a preamp you are closer to the master? If you like vinyl these guys will tell you their master is more realistic. 
There are many reasons to go thru a preamp:
1) if your dac/cd player is solid state, you might want to go thru a tube preamp that front end solid state amps
2) you might want to upgrade to an external dac, add a tuner, turntable phono preamp, etc...

Bring home a couple of good preamps and compare them to your direct connect setup

You may think at first that you're hearing a more transparent signal but a preamp properly matches the impedance between stages and provides more muscle (better dynamics) in the long run if it's a fairly good pre which it sounds like you have. You will get more dynamics and you won't be as susceptible to roll-off of high frequencies. The pre-amp is more than just a handy switching device, it provides an easy load for whatever you plug into it and outputs a lower impedance which will make the amplifier happier than if you just plug in a source device.

The first truly high end piece of gear I bought was a pre-amp. I couldn’t believe that change it made in my system. Without changing my midfi CD player or power amp, I was hearing a 3 dimensional sound stage for the first time, and a less brittle, much warmer and rounder sound. It excited me so much I rushed to purchase a high end power amp. It added bass authority and an even rounder more musical sound, but the soundstage magic was still a product of the preamp. I’ve been a preamp believer ever since, and would not consider using only a DAC with a volume control. That’s my limited experience.