Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
My Benchmark HPA4 preamp had the biggest positive on low level listening. It has a volume control with 256 settings and it is quieter than any other source component you will use with it. Once of the reasons I got it was for late night low volume listening. It has exceeded expectations on all fronts.
Speakers. You probably need a powered sub or two practically with most setups to sound best at low volume because our ear sensitivity drops off and we do not hear nearly as much as we do at louder volumes.

Equal-loudness contour - Wikipedia
mapman linked

https://en.wikipedia.org/wiki/Equal-loudness_contour

You want a system with automatic ’loudness’ circuit. It is completely misnamed, it is a ’low level boost’ to boost bass progressively as volume lowers, (boosts highs also, less importantly). Maintaining bass at low volume will maintain involvement with the music, otherwise it becomes background noise.

To compensate ONLY when listening at low volume. Raise the volume, eliminate the boost, otherwise the bass is too prominent as well as the highs will get screechy.

As you raise the volume, you want the circuit to automatically/progressively reduce/remove the ’loudness’ boost, back to normal for normal or louder.

This unit has a built in automatic fletcher munson circuit, and remote volume, remote balance, I wouldn't live without it.

https://www.ebay.com/itm/Chase-Remote-Line-Controller-RLC-1/254708363331?hash=item3b4dcd3043:g:huIAA...

Vintage receivers have ’loudness’ filters.


Loudspeakers. Good lightweight cone loudspeakers that are designed to sound good at low levels. Triangles come to mind. 
I lean towards speakers also, especially high efficiency with 10" or better woofers. My Cornwall 3's sound great at the low listening levels where they spend most of their time playing.