Which is the most important part of a stereo system?


My system consists of a pair of B&W 630's, an old Denon 50 watt reciever (DRA-550) from the mid 80's, a Marantz CD5004 cd player, and now a Pro-ject Debut Carbon DC Turntable.  I'm pleased with the speakers and the cd player and while the Denon sounds good it has some issues and I want to upgrade.

I'm planning on returning the Pro-ject TT and getting a higher end TT.  I'm also looking into getting a new amp pre amp with a internal DAC.

Is the source the most important? The speakers? 

Please Help!
klimt
Just in case nobody mentioned it already which they probably didn’t - the fuse, be it ever so humble. Another one nobody probably mentioned is electrical connections of all non-audio outlets. And one more nobody mentioned  - how many CDs are in the room.
Well, as often, valuable points of view. I'll go counter-culture. Without excellent power amp nothing will sound right with any source, preamp, speakers or cables. After that turntable itself, not tonearm or cartridge, and speakers - any order. Modest but basically good speakers will sing with great power amp, the other way around won't get you anywhere.
Another vote for the room.
It IS the most important single component affecting the sound by at least half or more of what you will hear.
It's also the most overlooked/disregarded.
The majority of us just keep focusing on the gear.
There's a reason someone came up with "audiofool"...
The room is the most important part. I’ve had a nice setup in a bad room and it sounded really bad. I would rather listen to a mediocre system in a great room over the highest of high end systems in a terrible room any day. The only exception I can think of might be dipole speakers which do a good job of “removing” the room from the equation. 
Post removed