I'm in the amp camp and have found big changes there, even within a manufacturers lineup. With my current spkrs, I've worked my way from the entry level amp to the top offering and my system has benefited with a bigger soundstage, more detail, more bass impact & definition, along with a smoother midrange.
Of course, w/o a good room, the best sounding system will sound like mud and that should be of prime concern before you start swapping gear. There are simple things you can do, the easiest & one of the most beneficial being spkr placement.
There are other tweak things you can do like room treatments and isolation that can help but even with the best room, poorly matched gear will sound, well, poorly matched.
It's important to only do one change at a time so you can keep track of where you're headed in regards to improvements.
Part of the fun in this hobby, at least for me, is changing the system around from time to time. I've kept the same system together for 5+ yrs on a few occasions & other times change things once a month. It all depends on what you want to hear.
One last thing-on salesman. When you find one that will work with what u have, listening to what you're looking for & then making suggestions, which include home demo's, then you've found a salesman worth his weight in gold.