Which Component Has the Greatest Affect on Low-Level Listening

Start with clean power feeding your Amplifier Perhaps a conditioner regenerator or as stable as you can get it into your system. Then as much power as you can supply with your amplifier.
I work from home and 75 % of my listening is low level
High level electronics will make good speakers sound great

Speakers with high sensitivity and paper cones usually sound good at low volume. I own a pair of Tekton Enzo 2.7 which are rated 98 db sensitivity and have 2 8" woofers with paper cones per speaker. Due to an upcoming move, I had to move the speakers into a small room, where I sit about 6 ft away and play at a low level most of the time. They sound good loud, soft or medium. The bass is surprisingly good at very quiet levels even without any eq boost and as long as you’re sitting relatively close. I also used to own Zu speakers which were similar in this regard - again high sens. and paper cones. In my experience, speakers with poly/plastic cones need to be cranked before coming alive. 

mapman

Modern Equipment without 'Lousness' filter:

Perhaps a sub with remote control would be nice, to boost bass to your ears for any volume/listening location/content. Or for full range speakers, remote tone, boost just the bass like you would boost the sub individually.

Remote, from my listening position is what I find so beneficial.

I listen to a lot of Jazz, and maintaining the Bass Player at low volume is what keeps the music involving at those low levels, maintains my awareness, while 

All the audio components through which electricity passes will have an impact on listening at any level.  

As an audio salesman for over 45 years, I will give you my answer...given countless times to clients...a "properly calibrated" loudness contour, and speakers that do not lack for extended bass response.  Start with bass shy speakers and they will shut down all the faster at low levels.  This is why the "high efficiency" response is counterfactual...high efficiency is achieved at the expense of bass extension in any given cabinet size.  True, with bass reflex a speaker designer can engineer a "bump" to mask the lack of extension, and that may make it more palatable at low levels, but then when played a "normal" audiophile levels, it will boom.
The "properly calibrated" loudness is rarely achieved.  If you have separates, and the preamp has a loudness function, then the amp MUST have input level controls to allow the system to be calibrated to the speakers' in-room response.  A "Variable Loudness" control, a staple of Yamaha products, is a fair compromise.  My 1962 McIntosh C-20 preamp has an excellent variable control, but hardly anyone is doing that nowadays.