Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
mapman Interesting answer about sub, I think trying a REL 9Ti with my stereo gear.  Thank you
Start with a very low noise room.  A rule of thumb is that your ambient noise level should be at least half of your average music level.  It's hard to get below 35dB ambient.
If I were to name a single component, I’d have to agree with yyzsantabarbara that a very well executed line preamp with attention paid to stable linear gain attenuation will have the most dramatic impact of fullness at ALL playback levels. 
Tone controls on the preamp to compensate for lower volumes. Took me years to realize this as for 30 years my equipment had no tone control.
A full range full frequency response speaker will have the most effect on low level listening. Also a full sounding amplifier will help at low levels.