The point is that with poor speakers your high end source may still sound poor and you won't hear the full benefit of your source. However with high end speakers which match your room you will hear the best your source can produce, good or not so good!
Everything matters but speakers shouldn't be the weakest link in your system.
Hence speakers first.