Which Component Has the Greatest Affect on Low-Level Listening

In my experience high sensitivity loudspeakers do a good job on low volume. At least my 98dB Ocellia’s do a very good job at low level listening. 

Start with clean power feeding your Amplifier Perhaps a conditioner regenerator or as stable as you can get it into your system. Then as much power as you can supply with your amplifier.
I work from home and 75 % of my listening is low level
High level electronics will make good speakers sound great

Speakers with high sensitivity and paper cones usually sound good at low volume. I own a pair of Tekton Enzo 2.7 which are rated 98 db sensitivity and have 2 8" woofers with paper cones per speaker. Due to an upcoming move, I had to move the speakers into a small room, where I sit about 6 ft away and play at a low level most of the time. They sound good loud, soft or medium. The bass is surprisingly good at very quiet levels even without any eq boost and as long as you’re sitting relatively close. I also used to own Zu speakers which were similar in this regard - again high sens. and paper cones. In my experience, speakers with poly/plastic cones need to be cranked before coming alive. 

mapman

Modern Equipment without 'Lousness' filter:

Perhaps a sub with remote control would be nice, to boost bass to your ears for any volume/listening location/content. Or for full range speakers, remote tone, boost just the bass like you would boost the sub individually.

Remote, from my listening position is what I find so beneficial.

I listen to a lot of Jazz, and maintaining the Bass Player at low volume is what keeps the music involving at those low levels, maintains my awareness, while 

All the audio components through which electricity passes will have an impact on listening at any level.