Regarding absolute phase/polarity, on recordings which are heavily multi-mic'd and are subjected to extensive post-processing and mixing (as many and probably most recordings are in recent decades), and assuming that multiple instruments and/or voices were recorded, as I understand it the finalized mix is likely to consist of a conglomeration of randomly phased sounds from each of the performers.
In those cases, while the two polarities may or may not sound somewhat different, it seems safe to say that neither polarity can be thought of as being correct. Maintaining absolute phase/polarity would seem to be meaningful pretty much just on recordings that were engineered with "purist" mic techniques (meaning just two or three mics), and minimal post processing. Such recordings would include many of the classical releases on RCA, Decca, Mercury, etc. during the early years of stereo.
Also, it wouldn't surprise me if in some cases where listeners perceive differences between the two settings of a preamp's polarity inversion switch what they are actually hearing is changes in the sound of the preamp itself, that result from the differences in the preamp's internal circuit configuration between the two settings.
Regards,
-- Al