Which is the most important part of a stereo system?

....and jeffvegas lives in 'a van down by the river'...

A highly armored one... *tease*  It's best to stay mobile with max'd cards....;)

There is no most important component. They call it a stereo “system” for a reason. My own approach is to figure out what the weak link is and address that next. It’s a never-ending project and the improvements get smaller and smaller but my ability to hear them gets better and better. 
That said,I can remember getting three big jumps in SQ over the past 20 years, that you might cogitate over before chasing boxes. 

1. A hospital grade wall outlet. A genuine WTF moment. No PC ever made as much difference as the outlet itself did.
2. Antivibration under the pre-amp. Iin my system it has mattered there more than anywhere else.

3. Perfect Path contact enhancer applied as the maker recommends it. An astonishing product, but you’ll never find it again. The maker died this winter and apparently the formula passed on with him. The entire community should be sad about that one. It turned out I purchased one of his very last tubes. 
but that is with my system. For someone else it might be something else. Tthere is no magic amp or pre or speaker out there that will transform a system. 
I could add a fourth — a well produced  recording. The better your system is,the more you are at the mercy of the production quality of whatever you’re listening to.

The room. 
I’ve head so-so systems sound wonderful in a great room and amazing gear in a poor room sound meh. 
Everything matters absolutely but the rooms acoustics make it all possible.  

The front end records,or digital is where  your music starts ,you can’t makeup downstream any shortcomings,therefor it better be optimumbefore going to the preamp section.

@audioman58 , yes I think everyone would agree that mastering matters.

It's virtually impossible to make poorly mastered records sound good - no matter what the size of your endless  audio money pit may be. 

We also happen to be fortunate to be living in a time when the gap between budget and high-end amplification and digital sources/converters/cables is almost indistinguishable too. 

What are we left with then?

Loudspeakers and vinyl playback. 

No one would argue that budget and high-end speakers and decks can be confused in any listening tests.

Even today's constantly improved entry level Rega/Pro-Ject decks (or any sub $500 loudspeaker you can think of for that matter) still leave considerable room for future upgrading. 

This hobby is far from dead, and we haven't even begun talking about Tube amps.

https://youtu.be/vCk12xzivp8