How do you determine how much to spend on speakers

Garbage in, garbage out. You cannot fix poor signals downstream. As a rule of thumb, add up what you'd spend on a CD player, tuner, turntable, and tape deck. Whatever that amount is, should be about what you'd spend on the speakers. A balanced system sounds far more musical and enjoyable than having too much invested in source, speakers, or amplification. 30% on source components, 30% on amplifier and preamplifier, 30% on speakers, 10% on wire (interconnect, power cords, speaker wire).
If you have a total of $5k and don't care for analog then you'd spend roughly $1500 on a CD player and tuner, $1500 on amp/preamp (maybe an integrated?), $1500 on speakers (including stands if they're not floorstanders) and $500 on all the wires you need.

$(speakers)>=$(power amp(s)); $(source+preamp)=$(power amp) than adjust it to your taste and listening needs...

Budget more on the front end and then find the best speaker, instead of the opposite way around. Speakers rely on what is fed to them. But best suggestion of all, you should listen to see what you like.

I initially spent 70% of my budget on speakers, and spent the balance on a receiver and cd player. I basically threw my money away on the electronics. And because my amplification and source weren't up to par, I never realized the full potential of my speakers. Now I'm up to my third rig, and finally upgrading the speakers.

My rec. is to spend as much on speakers as you "have too", but not at the expense of the other electronics. Even though "it's [$$ on speakers] an investment" it's silly not to let them [speakers] shine with good electronics. I sure wouldn't spend more than 50% on my loudspeakers. you can find $2-$3k speakers that will shine with $5 worth of electronics, if you're careful in selection. Look at the Vandersteen setups in a lot of hi-end shops. I think a saw a pair of Vandersteen 1C's (<$1k) driven by $5k in electronics. And it was a "good" setup.

And look what setup is suggested to people wit $1k budgets. 33% one source, 33% integrated amp, 33% bookshelf speakers. Listen for a while, then upgrade DAC or TT-stuff, or get better wires, then maybe add a musical subwoofer down the road.

So spend what you "have to" at this point. You'd upgrade later anyway, unless you are good at modifying/tweaking speakers.

I agree with Jeffloistarca and some others here. His breakdown is well balanced. Better speakers are also more revealing, so if you get great speakers and have cheap electronics, the great speakers will sound like crap, because they do a great job of reproducing the less than great sound of your electronics.

I am always reading speaker reviews in Audioreview where the consumer reviewer will say some speakers like B&W Nautilus sound like crap. Then they say they are not worth the money because you will have to spend alot on electronics to make them sound good. What they do not realize is, the crap they hear is the rest of their system, not the speakers.