Nothing is completely passive. The more revealing the circuitry, the more it exposes component 'flaws'.
Experience
with phase flip switches in active and passive balanced hardware often
left something to be desired in terms of routing and hence inaudibility.
Making both halves of a balanced circuit identical is EXPENSIVE. Add in tube vagaries and the probability of a sonic delta is fairly certain.
These statements are by no means universal truths :)
'Passive' refers to a component that does not involve a semiconductor or vacuum tube. Switches, sockets, capacitors, resistors, volume controls, transformers, wire and the like are considered *completely* passive.
Our MP-1 preamp was the world's first balanced line preamp for home use. The phase switch has been on it since its inception 30 years ago. Quite simply, operating the phase inversion switch is subtle- on multi-channel recordings you can't hear any effect at all. IMO whatever your experience was, it would seem to have been done with equipment that wasn't thought out all that well.
We use differential circuits for our balanced operation. This is not particularly expensive to do- our preamps are pretty affordable compared to the competition in the same class! It is true that each half of the differential amp is slightly different, but you might be amazed at how well they work- the key is a good constant current source, with which we get Common Mode Rejection Ratios (CMRR) in the neighborhood of 100dB, which is pretty good. In a nutshell, no 'sonic delta' when the inversion switch is used (unless playing a true stereo recording); but even if the differential amp is poorly balanced, its a simple fact that there is no 'sonic delta' anyway; you might want to think about why; hint: it has something to do with differential amplifiers :)