Most Important?

I would focus first on the proper amp, or integrated amp to drive my speakers. Now that doesn't mean that I'm spending more money on the amp than the source, but the amp/speaker interface is very important. A really great amp may or may not be very expensive, it all depends on how it interacts with your speakers.

A premium source is also a high priority. The real key is balance. A great source with a cheap reciever will not get you any better sound than a cheap source with a great amp.
Sorry, there is no real answer, only you can determine what works for you.

Following Tvad's example, right now my money (MSRP) is in:

1) Analog Source (table/arm/cartridge)
2) Speakers
3) Integrated amp
4) Digital source
5) Various tweaks, rack and isolation devices
6) Power conditioning/dedicated lines
7) Power cords
8) Speaker cables
9) Interconnects

Cheers,
John

Has anyone even considered the room in this thread? About 50% of the sound we hear is indirect--coming from the speaker interaction with the room. Putting a small investment in the room can frequently yeild results that could not be had for 10 times the cost in equipment.

Just a thought. (Yes, I'm biased on this subject)

Has anyone even considered the room in this thread? About 50% of the sound we hear is indirect--coming from the speaker interaction with the room.

Yes I did mention that after speakers, the room acoustics was probably the most important thing. I agree with you 100%

As an example, is the 3% second harmonic distortion of a loudspeaker less consonant with the fabric of the music than .3% IM in an amp?

Good point but 3% distortion in a speaker is amazingly good and 0.3% IM distortion in an amp is lousy. You are quite right that some forms of distortion are far worse than others though and I agree that IM is one of the bad kinds - great point there. FWIW, there is a good correlation between IM distortion and THD figures for amps...low THD generally means low IMD. I agree speaker harmonic distortion is fairly benign - you scored another point there - although high 3rd harmonic is fairly common in speakers and this is BAD. Also speaker transient response is usually abysmal and looks nothing like a real transient (amps are usually very good at this) - so there are lots of details to argue over, however I maintain that speakers are usually the most inaccurate device in any audio chain.