Which Component Has the Greatest Affect on Low-Level Listening

I lean towards speakers also, especially high efficiency with 10" or better woofers. My Cornwall 3's sound great at the low listening levels where they spend most of their time playing. 

Speakers make the biggest difference, followed by amplifiers.  Horn systems are particularly dynamic and engaging at low volume.  If you look at Japanese audio magazines, it is common to find serious audiophiles putting gigantic horn systems in their small homes and apartments.  Given the close living conditions and cultural emphasis on being polite and considerate, systems cannot be played very loudly, hence the desirability of horn systems.

I find amplifiers also play a big role.  Tube amplifiers, particularly low-powered tube amps, deliver lively and engaging sound at lower volume levels.  I find output transformerless tube amps to be particularly dynamic sounding.

Bass boost or loudness control is the right idea but powered sub is better. It provides extra muscle to provide the power hungry bass without taking it away from the rest. An amp driven less hard typically sounds better with lower distortion. Power demand increases exponentially as frequency lowers. It takes a lot of power for flat response to 20hz even at lower volume.  Especially in a larger room. 

To me super efficient speakers driven passively by say a Schiit Aegir 20w Class-A. are the way for low level listening, "IF" you can handle their colorations/distortions, I can’t, "especially" horns.

Cheers George

Music tends to sound thin at low volumes, that's why the Fletcher-Munson Loudness curve was developed.  Most higher-end stuff doesn't have a loudness control, nor does it have tone controls.  For me personally, the addition of the Schiit Loki allowed me to tailor the sound both for different headphones as well as for speakers at low volumes.  BTW, the suggestion for power conditioning and power cords is pure nonsense.