The external USB options might be a big improvement over your internal M-Audio card. I have no experience with the M-Audio card to say for sure, but in my research it seemed to me that none of the cards could touch the external devices. You'll need someone more technical than me to explain why. Examples would be the Waveterminal U24, Empirical Audio's OffRamp, or other products that convert USB to S/PDIF via an external box. The other option would be to go to a USB DAC, but since you like yours, there should be no reason to go with the former suggestion. I've had great results using iTunes + WAV and Apple Lossless + Waveterminal U24 to DAC. I'm betting the culpret is your card.
Marco
Marco