As many others have said speakers are probably the part that can change the sound the most. Still, an unbalanced system can sound worse than a slightly cheaper but well-balanced system. And changing just one component in a balanced system to another that is much better and maybe much more expensive will make the system unbalanced.
For example, a system with $100k speakers and a $1k amp might sound a lot worse than a system with $10k speakers and $10k amp.
If you have a well-balanced system and suddenly get a lot of extra budget to improve it it is probably best to replace many parts at once.