This sounds like a classic phase problem. Yes, try a mono recording if you have one, or a regular stereo one if not. Switch the positive and the negative on one (only) of your speakers. Perhaps you have purchased a (used?) pre-amp/you don't have the manual/ and it reverses polarity. Many do. In any case if it sounds better, with centered image and better bass, that's it.
If that doesn't get it and you've tried the above, go ahead and make speaker placement adjustments, move one closer/forward, angle one more than the other (this makes a big difference in centering), because your room is not perfect. Things don't have to be perfectly symetrical, you neurotic engineer.
(I know, we're all neurotic.)