Several factors come into play, both on the amplifier side and the speaker side.
The amplifier's distortion envelope is very important. Large amounts of second harmonic distortion are inaudible, but very small amounts of high order distortion can be quite objectionable. And a type of distortion called "crossover distortion" that occurs at the zero point with class A/B amplifiers is particularly nasty. Its usually masked at medium and high power levels, but is unmasked at low volume levels. If an amplifier's distortion rises at very low power, that's likely a sign of crossover distortion. Class A amps are free from crossover distortion, as are Class D amplifiers, and of course some Class A/B amplifiers are better in this area than others.
As you can infer from the above, loudspeaker efficiency can come into play. If the speakers are high efficiency and the amplifier suffers from crossover distortion, then the amp is more likey to be used at very low power levels where that distortion is most audible. This is probably one of the main reasons high efficiency speaker owners prefer single-ended triode amplifiers, as they are inherently Class A and free from crossover distortion, which under the circumstances would be far more objectionable than the high second order harmonic distortion they do generate.
Also, if a loudspeaker has drivers with significantly differing power compression characteristics (meaning they don't all increaase in loudness at the same rate), then the system has probably been voiced to sound right at medium to high volume levels and will have a tonal imbalance at low volume levels. Usually the tweeter compresses less than the woofer, so most common is a speaker that sounds dull at low volume levels and overly bright at very high volume levels. This is less likely to be an issue with high efficiency loudspeakers, and is a non-issue with single-driver loudspeakers.
Finally, I have read that some woofers have suspension systems that are non-linear at low power levels and so the bass is weak at such levels. Frankly I think it's more likely that weak bass at low power levels is a function of the ear's reduced sensitivity to low frequencies at low volume, but wanted to mention it just in case. Rubber surrounds are most often accused of being non-linear at low power levels.
As long as an amplifier isn't driven into clipping, I don't think it's how powerful the amp is per se that will make the most difference in its low-volume-level performance.
Best of luck to you,
Duke
