Air is non-linear, but this affects air pressure with respect to altitude. As long as your altitude stays constant while the speakers are moving in and out, you should be OK :)
Absolute phase *does* make a difference though- we have had a phase switch on our preamps for just that reason, since 1989. However it can be a bit of a trick- its best heard with 2-mic recordings. Multi-mike recordings tend to be quite confused. Also, you can expect that fully 50% of all media is in phase and the other half to be out of phase. This is because there are a vast number of variables between the microphone and the cutterhead (or CD master).
We had a well-known proponent of absolute phase turn up in our room at the recent CES (T.H.E. Show). He wanted to demonstrate to us the importance of absolute phase, and that some certain percentage of CDs were out of phase. As best as I can figure, he was not expecting that we would have a phase switch in our gear. So we put on a CD, then had us flip the phase. Now I knew the recording rather well, and I felt I could hear the difference (which nevertheless surprised me a bit) but interestingly enough, *he* did not hear it. One of the people in our room knew that we also had the same recording on LP, so we played that too. It was instantly better.
None of this went the way this guy wanted! He said to me 'this test does not work if the master tape is analog' and left the room. What poppycock! Now I think absolute phase makes a difference, but IMO there is also an awful lot of nonsense surrounding this subject...
