+1 Exactly.
If you want to look at another very respected amplifier design that has a residual waveform imposed on the output signal, look no further than the Berning amplifiers. These amps are excellent; they are liked by everyone who hears them. They employ a switching power supply at their output which is modulated (loaded) by the operation of the power tubes. The switching supply thus has the audio signal imposed on its output. The switching frequency is then filtered out, leaving the audio signal to drive the loudspeaker.
If this sounds familiar to those versed in class D amplifier concepts, it should because its a very similar idea!
So how is it that such a respected amplifier as heard by all comers gets a Murphy while class D does not? They both have a residual. The conclusion can only be that the residual is not harmful in either case.