Which is the most important part of a stereo system?

Lots of good advice here. Millercarbon probably offers the best. For the last several decades I've been through many different audio setups and equipment have learned that one single component or cable doesn't make or break the sound that you want (unless that component is defective).
One example.....I once had a pair of speakers that I really liked but the amp driving them was old and sub par by many respects. So, I bought a really good and more costly amp. The sound wasn't nearly as good (to me) as that from the old amp. SYNERGY is the key.
One single upgrade may improve your sound but everything must work together as a "team". If you have the chance to audition a component in your system and in YOUR room that is the perfect scenario. But that can be a daunting task. There are dealers out there that offer that.
Good luck to you. Remember......good sound is what YOU think it is.

Millercarbon probably offers the best.

Probably???!

As a manufacturer of audio components, I was a firm believer of the source because I felt that all tone and sound stage came from there first.  You know garbage in and garbage out.  My partner and I recently built a hybrid 6SN7 power amp - think Counterpoint but way more advanced and two our items to a buyer.  He loved the DAC the preamp and power amp.  He had them there for a few weeks and he concluded that the amp was where he would start first.  So away we go!

Happy Listening.

@klimt for integrated amps with dacs

Modwright KWI 200 hybrid with dac is $6200 new, but used is under $5000

Hegel h390 class a/b with dac and streamer functionality is $6000 new 

NAD m33 class d with dac and streaming and bluetooth functionality is $5500 new

All admittedly over your budget but only marginally and imho worth a stretch