Over the past decade or so, I've upgraded all my sources, electronics and cables and my Soundlab M1's have never failed to amaze me.
I suppose it starts with having speakers that do what you prefer and build around them. At least that's the way through the forest that has worked for me.
And I 2nd the:
1) If you don't have the source(s) right, it's probably unwise to upgrade either the pre or the amp.
2) Great line stages are more difficult to find & as a generalization, are more important than the amp, unless you have a very difficult to properly drive loudspeaker such as my S/L's.
3) At a certain level, all things matter equally.
Over the years, I've found that one needs to enjoy the journey rather than being focused on the arrival. The forests, mountains, and sunsets can be beautiful without being the "best" ever seen! The gold at the end of the rainbow can never be found. But it sure is fun trying. Enjoy....