Yes that is my experience - the cable transport itself does not matter too much, XLR versus RCA, for a home environment where runs are typically "short" (say 5 meters or less) and cables are shielded enough for the environment and signals carried.
HOWEVER, the circuitry engaged on either side of XLR & RCA I/O can vary a lot, and when the XLR path uses a symetrically "doubled" cricuit, and the RCA path uses only half of this, then it usually sounds better (sometimes significantly so) to go XLR, and often results in +6dB extra gain too.
Some circuits will take RCA input and invert / mirror it with an extra circuit (or opamp) to make a balanced signal, at slight perfromance penalty (versus the XLR input) - then run it through the balanced (doubled) circuitry all the same. Some will use an I/O transformer to do this on one of the other side (or both), adding the coloration of the transformer itself (which you may or may not like).
Then there are true differential inputs which are the only ones that don’t care what you feed into them. I have some high end headphone amps like this. You can feed in RCA or XLR with no difference in perfromance between them, and no penalty from extra inverter stages applied to unbalacned input sources. You can even send an unbalanced signal through its XLR inputs (e.g., via RCA -> XLR adapaters) and it’s fine too. Really cool; this is my favorite, though it is still not too common. Sometimes the amps like this are called Super Symmetry or SuSy, though that name is from Pass Labs. Pass Labs also has a different twist on its lower-level phono stages (Xono, XP-17), whereby the XLR *output* stage uses an extra inverter stage with +6dB gain. I haven’t directly compared them yet to determine if there’s a penalty to the XLR outs for this, but I can say both its RCA and XLR outputs sound excellent.
If you’re just "guessing" with unknown cricuitry, then usually the XLR path will sound better. But it’s not a guarantee.
On the cables themselves: I despise wrestling short, stiff XLR cables into place (pin alignment) - whereas an RCA conenctor can easily rotate-slide onto its jack from any angle! Audioquest’s later lines of XLR cables added a hard slippery nylon fiber net on their outer covers (for looks), which causes the strain relief to fail after several cycles of this "wrestling". This annoys me to no end. Their older models with soft cloth coverings are much much better here.