Where does fatique come from?

Usually from distortions occurring in the mid to upper frequency range.

One of the most common sources of fatiguing distortion comes from individual component's inability to accurately reproduce sharp transients and dynamics accurately. When these occur, the system cannot respond fast enough and fatiguing distortions occurs. Some may find sharp transients and extreme dynamics fatiguing as well even when accurately reproduced because these create the high pressure level variations in the air as sound is transmitted that can stress or fatigue the ear itself.

The problem is much more common with digital sources compared to analog/vinyl sources which is why I believe more find digital to be fatiguing than vinyl.

Planar and electrostatic speaker systems in particular ten to not move as much air and typically produce transients well and do not deliver in genreall deliver the extreme dynamics compared to conventional speakers using dynamic drivers. This makes them often an attractive option for those that may be fatigued by more dynamic designs.

As covered by folks above, fatigue is mainly caused by brightness especially in the midrange to high frequency range. Sometimes there are simple solutions that can ameliorate on this "fault" other than swapping equipment or speakers. Two effective way out is in speaker placement and room treatments. Room treatments will greatly enhance the sound system while eliminating any deficiencies such as sharp piercing treble(which results in fatigue) or bass boom in the low frequencies etc. Also, you might be surprised that a simple speaker positionining may eliminate fatigue altogether by a minimum or no toe-in speaker placement.

Maggies are well known for being "easy on the ear". They are not fatiguing at all, sound better the longer you listen, and you are seldom inclined to stop as a result of fatigue. These are planar designs that have very good transient response (low distortion there) and they do not move a lot of air compared to conventional full range dynamic designs, so dynamics are generally not as fatiguing either.

Omnidirectional speakers, like OHM Walsh designs, can also be comparatively easier on the ears as well because they do not blast all the sound and the associated high pressure levels in the air directly at you. They tend to fill the room with music instead, more like what occurs at a live, un-amplified musical performance, which enables realistic listening levels with less fatigue as well.

Yes, direct sound to the listening position is certainly not too good. With the drivers directed away from the listening position especially the tweeters, the ears will not be subjected to direct sound that may cause fatigue. One may also be able to crank up the volume higher without hurting the ear-drums with some proper speaker placement especially with minimum toe-in. In this sense, I suppose that omni-directional speakers may hold an advantage although I have no experience with Ohm Walsh designs.

BTW, this is an excellent question. The best way to a pleasant listening experience is to eliminate or minimize factors that cause fatigue. It is something that is not discussed as often as are things like tubes versus SS, vinyl versus digital, detail, imaging, etc.