My two cents, from a guy who commercially designs (and sometimes even makes money :-)) on electronics. But at the end of the day, speakers -- in my opinion -- make by far the largest difference. Speakers have many many opportunities to grossly distort the signal - flexing cones, resonating cabinets and baskets (the term for the driver frame), crossover trade-offs. PLUS !!! the simple fact that to reproduce bass and volume demands size -- and to avoid dopplering of the entire cabinet, mass and size, which cost money. Lots.
Now, several posts above point out caveats. It is possible to make an excellent, small monitor. In a small room, with chamber music, they may fit your needs and be had for 1/3 to 1/4 what a similar quality full range monster might cost. So there are exceptions.
Also it was pointed out that one must distinguish between "better" and "different". True again, but i’ll state that many speakers are just "better".
I am designing a new integrated amp right now. Probably my first "from scratch" design in 25 years, employing all new method that were impractical even 10 years ago. So i get lots of opporutnity to hear inexpensive ($3-4k) electronics with my "big" speakers as well as my usual costly stuff (prototypes of my best products from years past) on my smaller, cheaper, and yes, "worse" speakers. Clearly worse, BTW.
So i have over the years become more and more in the camp of "speakers are #1". Now, the sum of turntable, DAC, digital source and power supplies, amp, speakers may exceed it in aggregate, but the speakers ought probably take a big chunk.
One more caveat: its easy to make lousy sounding electronics. but its also possible to make GREAT sounding ones for a few Thou....but many sacred cows maybe slaughtered on the way.
G