It is really hard to engage with so many undefined terms like "phase distortion" being used, but based on your description, you aren't hearing change in relative time, so much as changes to frequency response. Comb filtering effects caused by different speaker positioning. It's an amplitude vs. frequency issue, even if it's a very tight set of them.
Best,
E
Best,
E