First off, let's assume you've narrowed down your shortlist to exclude loudspeakers that are obviously harsh or edgy, and now you're trying to choose from among a few very good-sounding speakers which is most likely to remain non-fatiguing over the long haul.
Listen to a wide variety of music, at normal volume levels, lower than normal, and higher than normal. What you're trying to do is ferret out sublte colorations that will become distracting over time, and also make sure the speaker's characteristics don't change drastically with volume level. I've made the mistake of taking on a loudspeaker line that sounded great at medium and high volume levels, but sounded dead and lifeless at low volume levels. Similarly, you want to find out if a speaker is going to get too bright or forward on loud peaks.
Here are two tests you can do that are in my experience reliable predictors of whether or not a speaker will become fatiguing long-term:
First, turn the volume down much lower than normal, down to near the threshold of audibility. The bass will pretty much disappear, which will tend to unmask midrange problems. Also, this will tell you if the speaker has good resolution of detail and nuance. Does it still sound richly detailed, lively, and tonally smooth (even though the bass will be missing)? If so, that's a very good sign.
Next, turn the volume up a bit louder than normal, and walk out of the room, leaving the door open. Listening from outside the room, all you can hear is the reverberant field. Is there a convincing illusion of live music happening back in there, or is it obvious that you're hearing a pair of speakers? Note that live instruments still sound convincing from the next room. Unfortunately relatively few loudspeakers get the reverberant field right, and in my opinion this is a critical factor when it comes to listening fatigue. Let me explain:
The ears localize sound sources in a reverberant environment by suppressing the directional cues from repetitions of the original signal (reflections) arriving after a very short time interval - .68 milliseconds, or the time it takes for a sound wave to travel the roughly 8 inch distance around your head from one ear to another. This suppression of directional cues from reflected sound is called the Haas effect or precedence effect. Now, how does your ear/brain system determine if a given sound is a reflection, or a brand new signal? It compares the sound with other recently heard sounds (dating back about 40 milliseconds) to see if it's a reflection. This is why I can speak to you from across a room and even though 90% of the sound power from my voice that reaches you ears is reflected sound, you can close your eyes and unerringly tell exactly where my mouth is.
Now when the reflections begin to sound less and less like the first-arrival sound, the ear/brain system has to work harder to properly classify them as reflections or new signals. And this situation arises when a loudspeaker's varying radiation pattern results in a significant discrepancy between the tonal balance of the first-arrival sound and that of the reverberant sound. The result is often a headache after about 20 or 30 minutes, even if the sound is initially very exciting.
Note that during the .68 to 40 millisecond interval when the precedence effect is suppressing directional cues, the timbre (tonal balance and texture) and subjective loudness of the sound is still being influenced by the reverberant energy. So assuming the first-arrival sound is tonally correct, it would be nice if the reverberant sound reinforced that correct tonality. The ideal is you want the reverberant field to sound like the direct sound with the room's acoustics superimposed on top, as this is what a live instrument would sound like in that room. Listening from outside the room lets you zero in on this reverberant field sound.
A loudspeaker that is exceptionally non-fatiguing is the Gradient Revolution. The Revolution is unusually free from distracting colorations and maintains an exceptionally uniform radiation pattern up and down the spectrum. Designer Jorma Salmi went to great lengths to get the reverberant field right. The top end can be a wee bit on the laid-back side, so a lively amplifier is often a good synergy. The Revolution retails for five grand, but used ones often go for less than three grand. Disclaimer - in case you haven't guessed, yup I'm a Gradient dealer.
Maggies tend to do a good job with the reverberant field, as do most full-range planars. You could pick up a refurbished pair of original Quad ESL's, the "57's", for well under three grand. And then there are conventional-appearing speakers that are designed to give a good power response - I surprised with how good the little Avalon Symbols sounded from outside the room. A conversation with designer Niel Patel revealed that he gives high priority to creating a tonally correct sound field, not just a certain desired on-axis response. Nope, I'm not an Avalon dealer, but may become one in the future.
Duke