Listening Fatigue

Pretty much agree with Sean,volume in a good system begs to be elevated.The highs for me do most of the wearing out,Bob

I agree it could be any element in your system but the speakers you choose and the format i.e. analog versus digital are the main culprits in producuing glare, hash, digititis, painful, shrill, harsh sound. That is what makes music fatiguing. Also agree it highly dependent on the listener preference, the volume (I like it loud but only for an hour or so ) I can listen to my analog set up at low volume all day. Therefore it is frequently inherently a function of your preferences. On the other hand, I wouldn't buy a flabby or bloated system just to compensate for the brighter, clean sound of a fine system.
You may want to try more relaxed systems which aren't going to everwhelm you, such as a pretty monitor on a good box, like Merlins with a tube amp and listen in a relatively small space at moderate/low volumes if hours of critical listening are your ultimate aim.
I employ several choices to meet my musical needs. If I want a rock concert experience I use big box vintage horns with relatively high power solid state in a huge room 30' x variable widths. I usually limit that to an an hour or so. If I simply want to hear good classical/jazz/acoustic its moderate volume using lower powered tubes and an analog source which I can listen to for an unlimited time or until I get bored with it, then I have to rock my soul.

Regarding mint604's assessment of acoustic suspension designs; a certain person with the last name of Dunlavy happened to design some of the best sounding speakers of the era that are still holding there own this very minute. I wish I had some Dunlavy's, like the SC-IV, SC-V or SC-VI. Dunlavy would disagree with your hypothesis, as do I.

mint604 "The sealed acoustic suspension speaker enclosure is the single WORST audio invention of all time"
This statement is utterly ridiculous. Many of the worlds best speakers available are sealed acoustic suspension designs. Dunlavy to name just 1.

I think the quality of the recording has a lot do with listening fatigue. I can listen to quality recordings louder and longer than I can with poor recordings, even if I really like the performers/music.