Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
I didn't count, but a guess the responses are 50/50.

I have been reading more since posting the question:

- horns (which I've never owned) seem to have good impact at low levels
- loudness (which I do have) also has made a difference

Maybe combining the two will produce good results:
Luxman's new integrateds
Klipsch Heresys or Cornwalls

Just thinking about some possibilities, since low level listening is about 90% of my listening from morning til evening.

Thanks for all the replies!
Higher efficiency speakers will allow your amp to go further and should help when bass is boosted by a loudness control as well. 
In theory.....proof is in the pudding. 🙏
No matter what amplifier I was using my Harbeth SHL5's and 40's always sounded great at low levels. All the Harbeth's I have ever heard have played very well at low volumes. Almost every review of any Harbeth speaker seems to mention this. From my point of view the speaker is more important than the electronics for a satisfying low volume listening experience. Good Luck!

Nor speakers, nor amplifier...

Listening at low volumes with  all details, imaging and dynamic first is related to the room, even in near listening, if you are in a small room....

My experience is this with my room....

My experience is not the negation of the importance of speakers and amplifier at all.... But the main factors are, if the gear is relatively good  to begins with,  will be the room acoustic, and the noise floor of the house and the uncontrolled speakers vibrations...

 Audio is NOT synonymus first  with electronic design, contrary to what most sellers affirm, but with acoustic controls and others controls dimensions first,  in most cases if not almost all....

 The room is then the main component to understand low level listening....

Not the loudness buttton.... 😁
No matter what amplifier I was using my Harbeth SHL5's and 40's always sounded great at low levels. All the Harbeth's I have ever heard have played very well at low volumes.
FWIW, I found the P3ESR to be superior to the M30.1 (which I currently own) at low levels.