Chicken, or the egg…
Source. Sometimes speakers. Depends on what I want to listen to.
Switch gear around a fair bit. Sometimes I want to listen to a SE tube amp - that will dictate speakers. Sometimes I want to listen to Class A SS. Then I have options w speakers. Sometimes, I want to listen to a high sensitivity speaker. Again, then I have options w amps. Decision gets made for me if I put low sensitivity speakers in or a low powered amp.
But as said above, it all matters.
If I have to choose, then source. Garbage in, garbage out.