"Makes me wonder what we're hearing though"IMO, either you're...
1) Not hearing what you expect to hear, based on extensive personal experience with the records tested
2) Hearing what there is to hear, based on the fact that you just played the record at the correct speed, and know which one is which
3) Hearing what you do expect to hear, based on your preconceptions about what effect will be wrought by a 1% speed change
...or some combo of the above. The acid test of this perception would be to use unfamiliar recordings and a helper, having them played back for you blind at both the correct speed and 1% off, to see whether you can consistently identify which playback is correct and which is fast or slow. I think this would probably be an exercise in futility though, maybe even at 2% in many instances, or even higher with some other types of music.