So I switched from XLR cables...

XLR is designed for professional use where cable runs can be very long and a lot of stuff is being connected and disconnected so the locking feature is essential. One of the first decisions I made long ago was to figure out XLR is unnecessary for home use. Beyond unnecessary, it doesn't even sound as good. As you just learned for yourself.

XLR sounds better than RCA  in my system.  If your system isn’t fully balanced XLR may not work for you. 

Hi,
if your equipment is balanced from input to output then XLR will most probably sound better, as it will have less noise and more dynamics. 
XLR connection does not translate to fully balanced design.