Which Component Has the Greatest Affect on Low-Level Listening


I'm looking to get out of HT 5.1 and go strictly to 2 ch.  I generally have music playing all day, often just sitting done to listen to certain tracks.  

I've read that "some" speakers don't sound "good" until high listening level, and also an integrated such the Luxman 505 don't sound good at low levels (is that the reason they include loudness?).

Which component, the amp or speaker, has the greatest impact on low level listening quality?
bgm1911
Which makes me ponder if maybe the Loudness button on my old NAD 3140 back in the early '80s didn't have a buffer-amp in the circuit?

There's a truth in what several are describing here, and I really appreciate Raymon's comments about convincing dynamics at lower volumes making the experience enjoyable.

I've heard Preamps and similar buffer amps (taking the upstream VC out of the equation) make such changes in a system. Fullness of dynamics, tone & spacial characteristics at all volumes. 
The preamplifier, doubtless.  Why?  Because unless it is forty or more years old, it lacks a "Loudness" control, which was specifically designed to solve this problem  It emulated the Fletcher-Munson curve in reverse according to the level set.  So you got all the frequencies regardless of level, if you wanted that.
Good speakers are #1. All the rest of what you have will work to the level of your speakers and not a single bit more. Spend a million bucks and your limit is still the speakers.