If you have a recording that was done with 2 or 3 microphones, the effect can be heard. The effect has to do with the fact that audio waveforms are not always symmetrical. Take a look at an audio waveform on an oscilloscope and you will see what I mean. So the theory goes that if the waveform has some asymmetry, that there is only one way to reproduce it right.
The problem is that 50% of all CDs and LPs are 'out-of-phase', so its useful to have a switch. I knew an audiophile years ago that always reversed both speaker leads in order to determine which way was right, and then marked his recordings... a switch is a lot handier :)