The most important piece in a stereo system???

Garbage in, garbage out. You cannot improve the source downstream, it only gets worse. Get the best front end components you can (Analog, digital, tuner, hell even cassette deck). Build from there; having the best speakers, amp, wires, or room won't fix a flawed signal generated right at the very beginning of the chain. Common sense in my view. ATB, Jeff

Speakers! Gee whiz.

Fmpnd is right on (in my opinion).
Great speakers with poor electronics sound terrible. I'd add that cheap CD players (<$200) are pretty listenable through all but the highest end spkr/amp setups. Therefore if I were to assemble a system bit by bit over many months (assuming I already have some sort of cheap CD and speakers ala circuit city) my order would be (I would buy in this order to ensure that at all stages I had a system I could bare to listen to):
1. Amp (this will set final budget)
2. Speakers (spend approx same as amp)
3. cables (spend about 1/10 of total system cost)
4. CD/DVD player. (spend approx 1/2 of cost of amp)

Matching is important, and I think the CD source has the least overall impact on the soundless even than the cables (again .. just my experience, no flames please).
Note if you want an analogue front end then I think the turntable/arm/cartridge is equally important to amp and speakers, and should be purchased first. In this respect I think that digital front ends are much more forgiving.

In theory, as Linn states the source is the most important. In reality it is a complicated interaction between the source material, all components and the room. However what I think is a better question is. How can I get the best sound for my budget? I actually assemble my system in my own strange method. I methodically try to, over time get the best of each type of component for my budget. For example When a pair of Aerial 10Ts were available at the right price my a/d/s speakers went, Linn LP-12with Lingo Bye Denon ,when I found a sonic frontiers SFCD-1 at the right price my Tandberg 3015a went. CJ Premier 14 bye bye CL35 MK III, 10B , goodbye Luxman T110 , and so on. The same goes for cables etc. Everything matters , but I think cables can be a bad investment if you go too far in a moderately priced system I stop at Monster 1500i m 1000i and kimber kable. To acheive the above I sell vintage audio as a hobby online to help trade up whenever I can. Lastly being that cost is a key issue for most people (not the reviewers of Stereophile), In certain systems there is a sensible proportion you should spend. Is the $ 20,000 Linn CD-12 great ? Yes but in a $ 25,000.00 system its a bad balance. You would be better off with a Rega or other sub $ 1,000.00 or less player etc. etc. I almost forgot dont forget about using good source material like good clean vinyl, 180 heavy remastered vinyl. MFSL cds etc. Enjoy!

That's a false dichotmomy Fmpnd, because no one is going to pair no-fi speakers with high-end source or no-fi source with high-end speakers. So, construing the question to assume decent hi-fi components all through the system, whether budget or expensive, would you rather listen to a NAD cd player through Eidolons or an Accuphase cd player through PSB mini-speakers?

I've made the weakest link argument before, since speakers are the least accurate component in a hi-fi system, and all speakers are colored in one way or another. But, that said, there are a lot of inexpensive to moderately priced speakers that are quite decent.

The biggest changes in sound for me, from mid-fi to hi-fi came with changes in amplifiers and cd players.

As much as I love my Harbeths, I have to disagree with my friend Charlie and say source, but amp is a close second.