"Like 99.9999% of the questions, it all depends on the component and cable. Last month I demoed an expensive highly regarded DAC at home for a week and found AEB/ESU was much superior to RCA eventhough the RCA digital cable is 3X more expensive. The only way to know for sure is try both in your system."
As long as the cable has proper impedance and reasonable capacitance the XLR design wins hands down. You can buy the best solid silver RCA cable half an inch thick and compare it to any cheap-o balanced cable you can buy in a music store and the XLR to XLR signal transmission will be superior. For short distances, like less than 15 feet, the difference is often not important or audible. Balanced cables use differential signalling, which puts the portion of the waveform below the reference voltage on one conductor and the portion above the reference on the other conductor, with a common ground. That's why you see three pins in an XLR connector.
At the receiver the signals are recombined at some point, and the combining process produces a phenomenon called common mode noise cancellation. This allows much longer cable runs without signal degradation, and that's why balanced cables are used for pro audio and microphones. Consumer audio tends not to use balanced audio because the circuitry is more expensive, the connectors are more expensive by far (except for perhaps WBTs), and for short runs, like I said, there may not be an audible difference.
The differential signalling advantage applies to analog and digital signalling equally, because on a cable it's all really analog.
RCA cables are "single-ended" and that's an inherently inferior signalling technique.
So, will a well executed RCA design with a reasonable cable be better than a poorly executed balanced design? Yes, of course. But otherwise balanced cables are the way to go.
