the dichotomy is not illusory. there are two ways to judge the merits of a stereo system, namely, equipment-based and listener-based.
a stereo system has an affect upon the listener. the stereo system can also be evaluated as to the usual criteria of neutrality, accuracy, transparency, resolution, etc. .
i believe both metods, the sound of the stereo system, independent of its affect upon the listener, or the affect upon the listener are valid.
many audiophiles feel that a listener's emotional response to sound is irrelevant as to the sound quality of a stereo system. such an approach places the stereo system as the point of reference.
i have yet to read a reasoned definitive argument rejecting one or the other position.