I didn't like the scratchy noise while I moving volume pot when I removed the dac output caps.
Sounds like you have DC at the pot... is the amplifier AC coupling downstream of the pot? A small DC offset in the low level signal will most likely result in pops and clicks from switches and pots. A large offset would likely be the result of a failed component or poor design/construction - that could result in distortion or component failure. Where you really don't want an offset is on the output of the power amp as you may damage your speakers.
DC blocking caps should be rated so that they are effectively a short at audible frequencies, so they have little if any impact on the sounds you hear. I'm not sure how much mileage there is in swapping out coupling caps as they are one of the more benign components in audio circuits. But if you really don't like caps then you could look for components that use DC servos instead - they provide protection from DC but are more complex and use more components which means they are more prone to failure.