And in some pre-amps that contain a phono stage (or in a separate phono amp), that stage (or amp) may be of inverted polarity while the line stage is not (or visa versa). What are you gonna do, reverse your speaker leads depending on whether you are listening to an LP or CD?
And since, as atmasphere points out, most recordings are made without respect to absolute polarity, album mixes are made from recordings (done on a 16 or 24 track---or more---multi-track recorder) in which the channels may be a mix of both inverted and non-inverted signals (some microphones also invert polarity! Some effect boxes---EQ, reverb, compressors, etc.---do as well) ). Of that there is no solution.
And of course, many multi-driver loudspeakers have one or more of those drivers in polarity opposite to one or more of the other drivers. Of that you DO have control---don’t own such a loudspeaker!