Since the signal contains positive and negative values, the speaker is SUPPPOSED to suck in some of the time and push out the other part anyway. Speakers don't just move from neutral/zero to some positive distance from the voice coil. They also move backwards just as much. That's my understanding anyway, based on the fact that the signal doesn't favor plus or minus (otherwise it would be a combination of DC and AC).
Picture a bass note of 40 Hz with no overtones. A pure, repeating cosine wave. While it's above 0 volts, it's pushing the speaker. While it's below it's pulling. If you were to switch polarity, all you'd be doing is inverting the waveform. So it's still pushing and pulling. The only difference is you start off in the opposite direction. So for 1/80th of a second at the beginning it would be different. And somehow this makes an audible difference!
There must be more to it...