Chesky has a test CD with sample jazz tracks from their catalogue. One of the teat track offers two pieces of music where a track is played in normal polarity and then in inverted polarity. With such side-by-side tracks you can see for yourself if polarity matters.
Speaker design plays a big role in whether polarity is heard. Designs that attempt to preserve phase relationship and timing of arrivals of sound from each driver will make polarity more obvious. If you cannot hear polarity change because your speakers muck up phase/timing relationships, fear not, many very good speakers are not good at this test.