Source vs Target: where is the weakness?

The source is the key. However, there are real world trade-offs. For example, speakers are harder to sell and ship so should you get great speakers first and hold onto them? Or, what if a really great amp deal came along? Should you then build the rest of the system around it? Since these kinds of situations occur all the time, the pragmatic answer is normally "it depends". That's what makes this hobby so much fun. All else equal, most would argue to start with the source and build it out as Kinsekd suggests (I say "most" based on past threads related to this question). I look forward to other responses.

I agree with Kinsekd - this is the path I followed for my own upgrade and it worked extremely well with very satisfying results. HOWEVER, it requires speakers that are sensistive enough to expose the strengths and weaknesses of upstream components, including cable. This leads us to the all important historical question

Which came first the chicken or the egg?

As long as we're on the subject, "What came first, the chicken or the egg?" Hard to answer! I like "Ozfly's" answer
"it depends." I personally think whole systems should be put together from the get go rather than mixing/matching this and that and hoping things work out. Easilier said than done! Well matched inexpensive components many times out perform and provide greater musical satisfaction than
mega-bucks ill-matched components. Tough question. A great, efficient loudspeaker (without a too low impedance) could pobably made to sound great with mid-fi source/amplification.

P.S. It's critical to have an overall philosophy or approach in mind regardless of how you go. To Kotta's point, the system is the key. For example, do you want to go the SET/efficient speaker route? Are you interested in near field listening? Listen to as many alternatives as possible before embarking on the journey regardless of where you begin.

Isn't the listener the target? Perhaps we should be upgrading ourselves! Easier said than done.