Probably the USB converter. This is mostly about lowering jitter, but USB has its problems as well. If you want to make an immediate improvement and want to stay with USB, replace the converter with a Berkeley USB converter. Maybe you can get one used on Audiogon.
Another option is to add a $599 Synchro-Mesh reclocker to the S/PDIF output of your MF USB converter. This will be the least expensive option. You will need a second high-quality S/PDIF cable like the Empirical Standard BNC cable with RCA adapters. This will lower jitter, but will not fix any issues related to USB. It should improve clarity significantly. If your Music Server has S/PDIF output, you can also use it here and eliminate the USB converter.
If you want to get the ultimate improvement in clarity, depth, width and detail as well as dynamics, change to an Ethernet converter.
You can go direct Ethernet by buying a DAC with Ethernet input or get an outboard Ethernet converter that will give you a really low jitter S/PDIF coax output, like the Interchange from Empirical Audio. Less than 18psec of jitter. This is not the inexpensive way to go, but you will have a world-class digital feed for any DAC. Interchange costs $2999.00 plus shipping.
Steve N.
Empirical Audio
We are masters of low jitter systems.