Which is the most important part of a stereo system?


My system consists of a pair of B&W 630's, an old Denon 50 watt reciever (DRA-550) from the mid 80's, a Marantz CD5004 cd player, and now a Pro-ject Debut Carbon DC Turntable.  I'm pleased with the speakers and the cd player and while the Denon sounds good it has some issues and I want to upgrade.

I'm planning on returning the Pro-ject TT and getting a higher end TT.  I'm also looking into getting a new amp pre amp with a internal DAC.

Is the source the most important? The speakers? 

Please Help!
klimt
I find that everything matters.

For me, the most important thing is the amp/speaker interface. Get it right and you are on your way to a system that can sing. Get this wrong and well, you may never get it right.
I own the 1210 GR...it’s a fit and forget audiophile TT.  It’s maintenance free and bulletproof.  Sound on mine with a Bronze Cartridge is incredible.  Buy it yesterday!
I do not have decades of experience, but I am into the thought that one should strike a balance in quality among all gear in the system, within a budget. If you have 5K and spend 3K on a turntable or a pair of speakers, then you're left with not enough money for your other sources, preamp, amp, interconnects, power, etc. So what good is an expensive turntable if you have crappy gear downstream; conversely, what good is a pair of expensive speakers if you have weak components upstream? If the budget is very limited, the more the need to be careful, even audition the gear, as one is most probably buying entry-level stuff. Another way of doing it is one could conceive of a really nice system even if it is out of budget at the moment. And then at one's own pace acquire the components one at a time. With this direction taken, if one wishes, a system could be built around a favorite pair of speakers. (I myself believe that upstream components should positively work to help the speakers sound their best.) It takes patience but it pays in the end. I built a system in 2 years and I'm happy to have planned my steps and waited.  It's another story for the rich of course.