I tend to lean toward the speaker side of things as this addresses the "voice" of your system, however you must recognize that there is a balance to maintain in maximizing your satisfaction of the results.
If you are trying to get the "best of the best" during the time of initial purchase (and if you can afford it), knock yourself out.
Otherwise the best advice that I can give is to make sure that the purchases you make are:
(A) Well suited to run together (no use running a low efficiency speaker on 5 watts of power)
(B) "Modular" (i.e., having room to grow on either the speaker side or amplifier side or both, depending on how you will grow your system)
(C) You *LIKE* the final results (at least for the time that you want to own them).
I usually keep my components for a LONG time since I try to keep expenditures to a minimum. This means that I look for equipment that I can truly enjoy even if I get hit by lightning tomorrow (at least while selecting wings and a (high end) harp, I could say, "Dang, wish I coulda got 'xxx', but oh well, at least I rocked the house with 'yyy'!").
On the other hand, it is "modular" enough so I can switch components without undue downgrade in quality (and resellability in case of the "oops" factor).
Well that's my 2 cents worth, YMMV.