2 different questions here. Which is most important (and the question above was re amps and speakers not source and speakers). If you dont think speakers are more important than amps, then you havent been paying attention. But that's ok, enjoy yourself and dont worry about what anyone else thinks.
Second question, which component should you spend the most money on? Different question. It could be, in theory, that cost-based and market economics make speakers a better buy among hi-fi components than amps or cd players, so you dont have to spend as much to get good speakers as something else.
My fellow Harbeth user, Professor Greene, who writes reviews of speakers for TAS, has recommended in TAS and elsewhere to spend most of your money on speakers and dont spend much on cd players or amps. He wrote a review of the late $2200 or so Carver Lightstar II amp, designed by our mutual friend Jim Carver, calling it a perfect amp. To Dr. Greene, that means don't bore me with any other discussions of amps. He would recommend the Harbeth Monitor 40 speakers, about $7500 list I think. For my needs, the Harbeth Compact 7, $2500 list, is a better fit, regardless of price. That's my favorite speaker, so that's all I need to spend on speakers. Now, because I think speakers are more important than amps, should I necessarily spend less for an amp?
Depends on how much good amps go for. (I think the answer is yes, I can get a real good amp for a lot less.)
The other issue, which wasnt the subject of this thread, should I spend less for a source component, like a cd player, than for my speakers because I think speakers are more important? Dr. Greene would say yes because he doesnt perceive a great deal of difference in the latest cd players. I actually spent more for my number 1 cd player than for my speakers because that's what I had to pay for the cd player I liked. If I were in the market today for a cd player, I'd probably spend less because inexpensive cd players have gotten a lot better.
So, if you are in the market and asking this question for a usefull purpose, I still think the answer is spend as much as you can to get the speakers you like, then figure out what to do with whatever money you have left to spend. If you have a lot of money left over and you end up spending more for your amp or source, then the hi-fi industry thanks you.