Which is the most important part of a stereo system?

It's all important, but speakers are going to have the most impact on how the system sounds overall. 

I have someone coming to buy a pair of speakers today and after putting them in my main system so that I could dem them, the difference between my current reference speakers and those was quite noticeable. 

Putting a much more expensive turntable or CD player into my system wouldn't bring it back to the level it was with the other speakers.  Putting my Harbeths back in the system will restore it to its glory.  

After the SOURCE it is the loudspeaker. 

You've expressed your opinion.  Shouting and repeating it are going to do what?

I’d say speakers then source then pre then amps

Easy, the plugs