Which is the most important part of a stereo system?


My system consists of a pair of B&W 630's, an old Denon 50 watt reciever (DRA-550) from the mid 80's, a Marantz CD5004 cd player, and now a Pro-ject Debut Carbon DC Turntable.  I'm pleased with the speakers and the cd player and while the Denon sounds good it has some issues and I want to upgrade.

I'm planning on returning the Pro-ject TT and getting a higher end TT.  I'm also looking into getting a new amp pre amp with a internal DAC.

Is the source the most important? The speakers? 

Please Help!
klimt
I do not have decades of experience, but I am into the thought that one should strike a balance in quality among all gear in the system, within a budget. If you have 5K and spend 3K on a turntable or a pair of speakers, then you're left with not enough money for your other sources, preamp, amp, interconnects, power, etc. So what good is an expensive turntable if you have crappy gear downstream; conversely, what good is a pair of expensive speakers if you have weak components upstream? If the budget is very limited, the more the need to be careful, even audition the gear, as one is most probably buying entry-level stuff. Another way of doing it is one could conceive of a really nice system even if it is out of budget at the moment. And then at one's own pace acquire the components one at a time. With this direction taken, if one wishes, a system could be built around a favorite pair of speakers. (I myself believe that upstream components should positively work to help the speakers sound their best.) It takes patience but it pays in the end. I built a system in 2 years and I'm happy to have planned my steps and waited.  It's another story for the rich of course. 
I’m agree with roberjerman. The source is very important, but all the little things are very important, cables to. Don’t forget the speakers: everything has to match..  and than finding like a puzzle, the best pieces. Sometimes a lot of work to find the right “click”. Enjoy your music!!!
Too hard to answer. Every component is critically important. Without one, you have nothing.
Here is my prospective on what's the most important part of a great sounding hifi system.
You can easily find excellent source, or amplifier, or speakers or cables.  However, there are not so many good sounding preamplifiers and they make enormous difference for the quality of sound, even if you only use digital source.