I just bought a preamp (Aesthetix Calypso) in large part because it has such a switch and I was tired of switching speaker cables to change the polarity. You won't hear much if any difference unless your speakers are polarity-coherent (many aren't) and have no or very minimal crossovers. Even then, many discs were recorded in mixed polarity and defy anyone to hear a difference.
Bottom line: As Kal says, don't sweat it. But once you do begin to recognize polarity differences, you can't NOT hear them. Or I can't anyway. Best to leave well enough alone. Good luck, Dave