Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
@andrewkelley

My 5.1 was Marantz preamp and amp with Paradigm speakers all the way around, nothing high end. I want to get out of HT and go 2 ch. I visit a couple local shops and realize my equipment is poor for 2 ch.

Sell all that and upgrade to some quality gear, but it's important that when I just want to play music with the family around, it still sounds reasonable - my Marantz/Paradigm was terrible ITR.
As someone who grew up in the late 60's through the 70's and had always been used to having hi-fi amps and receivers with tone and loudness controls, I find it ridiculous that many hi-end amplification systems have nothing but an on/off and volume control. If you want tone control, you need to purchase yet another piece of equipment to add in the chain. Still others say tone can be achieved through use of certain cables. IMO- which is clearly "old school", having an amp or receiver with tone and loudness controls is a no-brainer. Why wouldn't you want to have more options to tailor the sound to your ears as opposed to fewer, or none at all? Don't want that much bass, mid or treble, cut them back, or run them flat. Many receivers have a switch to bypass the tone controls altogether. Nearly all recordings will differ in some way or another, through microphones, analog vs digital, studios, producers, mixing, mastering, you name it. The tone controls allow the listener to very easily make changes tailored to their own enjoyment.
Higher efficiency speakers are far better at low volume levels than power hungry higher spl speakers. You can overcome some of it but there are physics behind it as well that you can not overcome as well.
I would look for a good class A amp that will not change over to A/B until 5 watts. Low level class A sounds really good.
I’ve had very good results at low volume, with 97db horns and about 4-5 watts (300B).