@jimmy2615 - Your experience is a perfect example of "everything matters" and doesn't make any logical sense unless you assume that there are other unknown factors in play.
Any difference in sound should be expected to result from variances in your components (very minor), unbalanced room acoustics (potentially very significant unless you have a nearly symmetric listening room), and imperfections in your hearing (likely significant). If all of these factors were truly identical, then switching channels would create a perfect mirror image of the resulting sound. If you could visualize the sound waves as lasers, you could take a picture looking down for one setup and then looking up for the reverse setup and the two pictures would be identical.
As you noted, hearing a familiar recording backwards is a significant change and the more "stereo" the recording is the harder I think it would be to make any meaningful comparisons. I have a particular recording that has a guitar so much in the left channel that I've actually disconnected that side just to see if it was heard at all on the right channel (as expected it was). I happened to catch a local dealer with an interconnect connected backwards on one end with this song. They seemed confused that I was so adamant but did discover the L/R reversal upon inspection.
