- There are innumerable stages in the recording chain that may or not be polarity correct.
- In a multitrack recording, there may be tracks [drums, guitar, bass, etc.] with correct polarity and tracks inverted.
- Selections recorded in one studio may be inverted relative to another studio on the same album. Compilation albums are notorious for this failing.
- Some loudspeakers invert polarity between drivers.
- Almost every device in the chain introduces phase distortion, i.e. a time shift between fundamental and harmonics. Coupled with loudspeaker polarity, this can make it impossible to achieve correct polarity across the audio spectrum.
Given the above, it is almost impossible to use most recorded material to determine polarity.
In decades past, I marked LPs as to preferred polarity, but now I just enjoy the music.
Bottom line, set it to how it sounds best for the majority of material.