To me, the source is somewhat discrete relative to the rest of the system. A crappy source will sound crappy with whatever follows. However, the speaker/amplifier relationship is much more symbiotic, as several here have said. I believe somebody mentioned matching the speaker and the room but I haven’t seen anyone yet mention the relationship between the type of music listened to and the speaker. Sure a good speaker should sound good with most music but not every speaker can handle the power to fill a large room to believable levels with hard rock music, as one example. Also, speakers can alter tone as not every speaker projects a flat frequency response so some offer a presentation that seems to add a bit of warmth while others can be perceived as providing greater detail. In addition, loud is not enough for some types of music or for some listeners where there needs to be a feeling of weight and power behind the sound, as it is in real life. Conversely, not every speaker sounds good, or realistic, at low volumes. Having subs, or not, can also make a big difference.
All that said, I would start by finding the right speakers for my room, music, and listening preferences, then match an amplifier that can drive them convincingly, and finally work on improving source components. I view the preamp as sort of between the amplification/speaker side of things and the source, as in my experience the right preamp for a system can make it sound wonderful while the wrong preamp can be similar to ruining your favorite dish by adding the wrong spices or flavorings.