As an example, is the 3% second harmonic distortion of a loudspeaker less consonant with the fabric of the music than .3% IM in an amp?
Good point but 3% distortion in a speaker is amazingly good and 0.3% IM distortion in an amp is lousy. You are quite right that some forms of distortion are far worse than others though and I agree that IM is one of the bad kinds - great point there. FWIW, there is a good correlation between IM distortion and THD figures for amps...low THD generally means low IMD. I agree speaker harmonic distortion is fairly benign - you scored another point there - although high 3rd harmonic is fairly common in speakers and this is BAD. Also speaker transient response is usually abysmal and looks nothing like a real transient (amps are usually very good at this) - so there are lots of details to argue over, however I maintain that speakers are usually the most inaccurate device in any audio chain.