Electromechanical components (cartridges and speakers) will always have the greatest effect on sound. And of those two, speakers, having to also transduce between mechanical motion and air motion (plus the unknown room interactions) have the biggest variability in sound reproduction.
Simple thought experiment.
Which will sound better:
1) $1000 integrated amp + $10,000 speakers
OR
2) $10,000 integrated amp + $1000 speakers
If you think #2, then you have been brainwashed by the audio mags.
Always spend your money on speakers first. Buy the best you can afford. When time and money permits, you can try better electronics. You may be able to the tell the difference or not, and you can then make an informed decision whether or not you should make the upgrade. If your speakers arent good enough, then you will have no idea if swapping out the electronics or front end components is worthwhile.
For car racing, you can talk about suspension and motor and brakes, blah, blah, blah. In the end, its ALL about the tires.
For audio, its ALL about the speakers.