How do you determine how much to spend on speakers

Main thing is to make sure speakers are not undersized relative to the room. Larger more expensive speakers typically with larger and/or multiple drivers are needed for good full range sound in larger rooms.

IF you do not care much about extended and balanced bass levels, you can get by with something smaller.

There are even some larger monitor style speakers out there that may be up to the task, but you will often pay a premium for those and rightfully so.

There are many schools of thoughts on this subject but I would spend twice the amont on [2] speakers that I would spend on amplification.
Same principle applies to source components, again I would spend twice the amount on [2] speakers.
With a $5,000 budget that roughly works out to:
Amplifier: $1,200
CD or Turntable: $1,200
Speakers [2] $2,400
with $200 left for decent cables.
BTW, you can get a damned good system with a $5,000 budget if you do your research, shop around and yes, haggle. Always be prepared to walk away...

I would recommend spending 50% on good used speakers. Spend 25% on a good integrated amp, 25% on a CD/SACD player, and then take the last 25% that you didn't tell your wife about and buy a tuner and some decent cables. I realize that puts you at 125% of budget, but like public works projects, the term "budget" is nearly meaningless when it's applied to a stereo system.

By looking at my checkbook.

I have done it before. At mid end HIFI. Out of all the equipments, the speakers can make the biggest difference. Go and listen as they all sound different! Try to get as full range as possible as oppose to those refined monitors or you will end up addicted only to vocals. Save up next for the best amp to fit power to your speakers. Source doesn't make huge difference until you get into super hi-end. At that time onward, the source becomes the most important and only analogue can fill the bill.