You have to get good cables in both cases.
For optical I use DH labs - it's about the fasted out there (150 mbs) and I found it outperformed a higher priced cable from Chord
For digital coax with RCA plugs the RCA's have to match the impedance of the cable otherwise transmision is affected.
I've used Van den Hul coax cables and they perform very well.
No idea about XLR - so can offer no opinion
With good cables I have not found any difference in performance
I'll qualify my comments with - I've only tested coax up to 16/48 kHz - but have tested optical up to 24/96 kHz
I currently stream 24/192 via USB (asynch) from imac to Schiit Bofrost DAC.
Regards