Class D, good for powered subs and car stereo amps, and that’s about it. All my opinion.
I agree that a good number of older class D amps were not all that musical. I heard some early examples that I would have thought were a joke were it not for the price.
Class A relies on the bias setting and operating point of the output devices such that you get the maximum linearity out of them. This is important to keep distortion down and for getting a nice first watt.
The advantage of a class D amplifier is that controlling distortion isn't based on the output section. This is a tremendous advantage. At this point its more about how linear the encoding system is.
In a class A circuit, the linearity is important because in most cases, the amount of feedback that can be applied to the circuit is often limited by frequency poles (like coupling capacitors; in engineering terms, this idea is called 'phase margin'), which if exceeded, will cause the amplifier to oscillate. The other problem is that the application of feedback causes distortion which manifests as IMD and higher ordered harmonics through a process called 'bifurcation'. The ear interprets this as harshness and brightness, as it assigns a tonality to all forms of distortion. The ear is also particularly sensitive to the higher ordered harmonics as it uses them to sense sound pressure.
These aspects of using feedback have been known for a long time- Norman Crowhurst wrote about them in the late 1950s. Now you can get around the distortion problem by running enough feedback, but in most class A amplifiers this isn't practical since the amp will probably go into oscillation; you need in excess of 35dB for the amp to clean up the distortion caused by feedback.
Class D amps have a means to get around this problem. You can run 35dB or more of feedback with the expectation that the amp will oscillate- in fact its encouraged. The oscillation is used as the switching frequency, resulting in a fairly simple circuit that has low distortion and lacking the higher ordered harmonics that causes harshness in so many amps.