Best small, desktop speakers, $500 or under (new or used)

I’ve heard Vanatoos multiple times and was impressed every time.  Every one who heard them at the shows were really impressed too.  They do sound good, have ridiculous bass and play loud.  The vanatoo would also smash A2+’s.  
Remember, I bought the B2 so I’m not trying to say they are crap, they just aren’t in ghe same league at all

 Zdac, still not sold.  Looks like a lifestyle product with voice command.  My 15 year old nephew who is Apple and Alexa fan boy would buy it.

For $400 if I was looking for something like that, I’d buy a Peachtree Deepblue.  
Have you heard a Deepblue?

Another +1 for AudioEngine.

From what I understand, upgrading the crossover makes a marked difference to their sound quality?

Yes- replacing the non-polar electrolytic in the crossover with a film cap, which is easy to do.

The Optimus Pro LX-4 looks interesting too. I'll look at how to mod them.

See if you can find the Pro LX-5- its a better speaker in nearly every way. The only problem you run into with them is that there is rear-firing information coming from the tweeters, so the speakers have to be a bit away from the wall behind them for best results. But they are fast and quite smooth, although at this point the woofers will have to be either replaced (replacements are easy to find) or the foam surrounds on the woofers replaced (foam surround kits are easy to find too).

Type of music preferred, classical, jazz, but rock too. I.e., varied.

Speakers don't have taste in music and no-one has found a way to make a speaker better for a certain genre, so this is no worries.
Now the Rogers LS35/a and all the recent variants (Spendor, Harbeth, etc.) should not be ignored! This speaker is very musical and designed for nearfield listening.

Another vintage speaker that is a lot of fun is the KLH Model Nineteen. This is the smallest KLH of the original series designed by Henry Kloss and has the distinction of being one of the few he did that is not a sealed cabinet. Finding a set in good shape is hard, but this speaker is very fast and cohesive, as it has no crossover and relies on a pair of full-range high efficiency 4" drivers.

None of the speakers I've mentioned so far need any serious power- 5 to 10 watts is plenty!

Hold out for a used pair of Silverline Minuets — you wont be sorry.  And I’d cross the Clairaudient speakers off your list.  They’re awful.  I had the 1+1 on demo and were the worst speakers I’ve ever heard.  Took them to a buddy’s house to hear them in his system and he laughed at how bad they were.  Rolled-off, muddy highs, hollow- sounding mids, and no dynamics whatsoever.  IMO someone at TAS got paid for a positive review because these things are awful.  Did I mention they’re awful?  Anyway, hope this helps, and best of luck in your search.