Which Component Has the Greatest Affect on Low-Level Listening

Higher efficiency speakers will allow your amp to go further and should help when bass is boosted by a loudness control as well. 
In theory.....proof is in the pudding. 🙏

No matter what amplifier I was using my Harbeth SHL5's and 40's always sounded great at low levels. All the Harbeth's I have ever heard have played very well at low volumes. Almost every review of any Harbeth speaker seems to mention this. From my point of view the speaker is more important than the electronics for a satisfying low volume listening experience. Good Luck!

Nor speakers, nor amplifier...

Listening at low volumes with  all details, imaging and dynamic first is related to the room, even in near listening, if you are in a small room....

My experience is this with my room....

My experience is not the negation of the importance of speakers and amplifier at all.... But the main factors are, if the gear is relatively good  to begins with,  will be the room acoustic, and the noise floor of the house and the uncontrolled speakers vibrations...

 Audio is NOT synonymus first  with electronic design, contrary to what most sellers affirm, but with acoustic controls and others controls dimensions first,  in most cases if not almost all....

 The room is then the main component to understand low level listening....

Not the loudness buttton.... 😁

No matter what amplifier I was using my Harbeth SHL5's and 40's always sounded great at low levels. All the Harbeth's I have ever heard have played very well at low volumes.

FWIW, I found the P3ESR to be superior to the M30.1 (which I currently own) at low levels.

Speakers with big a$$ woofers sound great down low...