Whenever I play a new CD (my system is single source CD) I listen with the polarity both ways to determine which is right for that recording. Often, that setting works for all the tracks, but sometimes it is mixed, on compilation albums, for example. Once I have determined the best sounding position for that CD (or individual track), I mark it with a red or blue dot sticker.
My understanding of, and experience with, absolute polarity is that you want to match the polarity of the microphones that originally captured the sound, regardless of how many times the polarity may have been flipped between them and the sound coming out of your speakers. The reason for this is that there has never been a universal standard for wiring mics, so it could be either way. If you don’t have a method for changing the polarity of your system, then the odds of it being right (ie best sounding) for any given recording are about 50/50. In simplistic terms this means that half of your music collection will never sound as good as it could. This correlates very well with my experience, as roughly half my CDs sound better one way, and half the other.
Of course, this assumes that all the original mics were wired the same way. This may not be the case, especially with multi-track recordings. Even with these recordings, though, in most cases, I have found one setting preferable to the other.
A few components (mainly preamps) do have an absolute polarity switch (sometimes improperly labeled as “phase”), but most don’t. Without one, you will need to reverse the speaker leads at your amp, in order to switch polarity. Not exactly an easy or practical method. Implementing a polarity switch is relatively easy, if you have transformers somewhere in the signal path. If not, it gets a lot more complicated. My amplifier (Antique Sound Lab Tulip 2A3 SET) has input transformers, so I had a friend help me add a polarity switch to it years ago. Eventually, I hope to acquire a Music First Audio passive TVC pre with polarity switching, freeing me to upgrade my amp. Some may not hear a difference switching polarity, but a polarity switch is not something I want to live without in my system.