On this one I have to unfortunately disagree with you both. In my experience somewhere around half of all recordings are polarity inverted. While that means, that it doesn‘t matter which way you connect your speakers, the actual playback will be out of polarity about half of the time. That‘s why having a polarity switch in the digital domain is so highly desirable. It is actually an advantage of digital over analogue to be able to correct this, and: yes, the differences on a decent system are very audible.
BTW: @mahgister‘s explanation as to why is by a country mile the most lucid I have encountered on the subject.