I agree it could be any element in your system but the speakers you choose and the format i.e. analog versus digital are the main culprits in producuing glare, hash, digititis, painful, shrill, harsh sound. That is what makes music fatiguing. Also agree it highly dependent on the listener preference, the volume (I like it loud but only for an hour or so ) I can listen to my analog set up at low volume all day. Therefore it is frequently inherently a function of your preferences. On the other hand, I wouldn't buy a flabby or bloated system just to compensate for the brighter, clean sound of a fine system.
You may want to try more relaxed systems which aren't going to everwhelm you, such as a pretty monitor on a good box, like Merlins with a tube amp and listen in a relatively small space at moderate/low volumes if hours of critical listening are your ultimate aim.
I employ several choices to meet my musical needs. If I want a rock concert experience I use big box vintage horns with relatively high power solid state in a huge room 30' x variable widths. I usually limit that to an an hour or so. If I simply want to hear good classical/jazz/acoustic its moderate volume using lower powered tubes and an analog source which I can listen to for an unlimited time or until I get bored with it, then I have to rock my soul.