I think you hit on the critical aspect - the need to build a system that works well together. I remember the days when the reductio ad absurdum folks suggested (tongue in cheek, one hopes!) that the best way to build a system was with a Linn LP-12 and an AM radio and then to upgrade from there. That's obviously absurd.
But I do generally agree with the concept that the better your source(s), the higher the ceiling for the sound of a system. We used to do a demo where we'd play PSB 300 speakers ($300/pair in the mid-1990s; stand mounts) with a Naim CDS-82-250 and then dare people to guess how much they cost - most guessed around 3k, thinking they were the ProAc Response 2. The challenge I now have is upgrading my source - it's an Exposure 3010S2 CD player, which is terrific, but it's going to be very difficult/expensive to upgrade it in a meaningful way that still synergizes with the rest of my system.
At the other extreme, I don't think it makes sense to put, say, a $8k speaker with a NAD or Denon or Marantz or whatever receiver - I think you're going to leave a LOT of that speaker's performance in the box, so to speak.