It doesn't matter, so long as you connect positive on the amp to positive on the speaker and negative on the amp to negative on the speaker. Kind of like jumping a car battery, but less dangerous if you mess it up.
If you do mess them up, it can sound bad. I have no idea why. Since the signal is AC, it's negative voltage some of the time and positive voltage some of the time, so it shouldn't make any difference. But I have heard them connected wrong and for some reason it does sound terrible.