First, a speaker must have excellent low-level resolution. This is primarily a power-to-weight ratio issue, so high efficiency speakers do well here (very powerful magnets), as do electrostats (ultralight weight diaphragms). The suspension system comes into play as well; some suspensions exhibit hysteresis (slow return to rest position) that can reduce low-level dynamic contrasts.
Next, a speaker must have fairly smooth midrange to sound good at low volume levels. At normal volume levels prominent bass can mask midrange peaks and resonances, but very low volume levels can unmask these anomalies. The ear's sensitivity is weighted more towards the midrange at low volume levels, so glitches in that region aren't going to be masked by the bass.
Then we want the tonal balance to remain correct at low volume levels. We'd like to think that the woofer and tweeter both get loud at exactly the same rate (3 dB for each doubling of input power), but in reality that's seldom the case. The vast majority of drivers exhibit power compression of some kind, with the result being that a doubling of input power gives a less than 3 dB increase in loudness, and conversely a halving of input power gives a less than 3 dB decrease in loudness. Typically woofers compress more than do tweeters, so if a two-way speaker is voiced to sound right at say 80 dB, then up at 100 dB it will sound bright (tweeter louder than woofer) and down at 60 dB it will sound dull (tweeter softer than woofer). Single-driver speakers inherently do a fine job of retaining their tonal balance at low power levels; unfortunately many single-driver speakers have a somewhat thin tonal balance that leaves the bass completely out of the presentation at low levels.
Finally, at low volume levels good dynamic contrast is especially important, as we must rely on dynamic contrast to breathe liveliness into the sound because by definition we can't do it with impact. Dynamic contrast (lack of power compression) generally correlates very well with efficiency; in other words more efficient speakers are likely to have better dynamic contrast at low volume levels. The crossover seems to play a role here as well; this may be overgeneralizing, but crossoverless or simple-crossover speakers seem to me more likely to sound good at low volume levels.
Speakers that sound especially good at low volume levels include models from Quad (especially their original ESL, the "57"), Omega, Cain & Cain, Classic Audio Reproductions, Sound Lab, and Edgarhorn. These are all either high efficiency horn or single-driver speakers, or full-range electrostatics. Disclaimer - I peddle some of these. No doubt there are other speakers that sound quite good at low volume levels, but I'm just mentioning these as examples that I'm fairly familiar with.
In my opinion, if a speaker sounds good at very low volume levels, that's an excellent predictor of long-term fatigue-free enjoyment. I recommend that people listen at very low volume levels at some point in their auditioning process, because this will reveal a lot about a loudspeaker that can otherwise be overlooked in the excitement of how good they sound when cranked up. In particular, listening at low volume levels will reveal how good the low-level detail is, and that's often what separates the great speakers from the good ones. Also, midrange problems are often more easily highlighted at low volume levels. And if the speaker's tonal character changes markedly when you turn the volume way down, chances are it also changes markedly when a very loud peak comes along.
If I ever bring out a loudspeaker of my own, sounding good at very low volume levels will be a high priority.
Duke