IMD just does not sound at all like music. It is the complete opposite to harmonic distortion such as you get from mechanical drivers and clipping of tube amps (which can sound pleasant and may be totally indistinguishable from actual music...in short a matter of taste)
IMD happens when you have two signals intermodulating...say for example you have a woofer which is playing a kick drum or bass guitar playing with copious amounts of energy required at 85 Hz. Inevitably any transducer has inertia or mass (recall damping factor) and the amplifier is always fighting to keep the cone in the correct position. The effort placed on the amplifier is taken up by the power supply. Slight variations in power supply voltages due to an over demanding draw on current in the LF may affect reproduction of higher frequencies by the amplifier. (Typical challenges for an amp would be a reflex port designed to increase bass extension that causes a huge and sudden drop in impedance at a specific frequency...worse case might be a port at 85Hz on a small monitor with amazing bass sound or that reviewers describe as sounding like a bigger box speaker!)
The net result is that the simulataneous reproduction of the 85 Hz signal and a 1000 Hz signal (with natural harmonics of 2000 Hz etc and up) may cause the amp to output 1000-85, 1000+85, 1000-(2*85), 1000+(2*85) Hz etc. These sounds are totally unrelated to anything musical and sill stick out like a sore thumb (even if they are 20 db or more below the main signal levels)... at the very least they raise the noise floor but more often then not it means the mid range lacks clarity or sounds positively distorted in extreme cases. Simply turn up your amp until it clips heavily and you get the idea (in the extreme of course).
Better mid-range clarity, less harshness and an effortless sound is how I would describe a reduction in IMD distortion. (It is probably the reason that huge amps with massive power supplies are so popular even with modest speakers. It may also explain why horns are popular and known for dynamics and clarity - as they are generally an easier load on the amp. It may also explain why active subs are popular....by taking the heaviest load away from the main speakers' amplifier you get better clarity in the mids and there is a tendency to play music louder because it sounds sweeter or less harsh even at higher levels.)
Unfortunately, a single driver also suffers from amplifier induced IMD as the amp is still driving the sound across a broad range of frequencies and therefore difficulties caused by an impedance drop or difficult load at some frequencies may cause amplifier strain and IMD distortion in higher, more critical frequencies of the mid range.
However, any speaker that presents a very easy load to the amp such as a compression horn will likely benefit from less IMD.
Speakers with extreme bass extension (tuned reflex ports for bass enhancement) usually provide a difficult amplifier load and simply trade off great bass performance for less mid range clarity. This may explain why some veterans in the audiophile world will shun passive speakers with amazing bass extension down to 20 Hz in favor of a better mid range sound with less bass...there are two camps it seems on this particular point and a single driver limited range speaker is definitely in the camp of prefering better mid range sound at the expense of extremes in sound reproduction. An active three way is essentially like having three single driver speakers except that each of the three drivers operate over a very much narrower frequency range where reproduction may be optimized and where amp stress in music reproduction at certain frequencies (especially LF) will remain constained (through active filters and physical separation from other drivers) in a much narrower band; the end result is that it is much less audible (none of this is rocket science and this has been well known for decades, however, active speakers remain mostly relegated to professional applications so the "cons" continue to out weigh the "pros" in the audiophile world)
