Is there actually a difference?

@fleschler We're actually saying the same thing.  Shaw's point (as I understand it) is that a speaker needs to be matched with an amp that can drive it without significant alteration in the amp's output. A 20 watt tube amp and a 250 watt solid state amp will sound different on a speaker of moderate sensitivity with a difficult load.  However, that difference is due to the tube amp's interaction with the speaker's impedance curve. If both of the above hypothetical amps were paired with a speaker that is high sensitivity and an easy load, they will both sound remarkably similar in blind, level-matched tests.  

You say "My speakers sound vastly different as do many high end speakers based on the amplifier's characteristics. Difficult load speakers due to impedance, phase angles or sensitivity present all types of challenges for amplifiers."  

Agreed!  But the idea is that an amplifier that can meet that challenge (and why would you want an amp that couldn't?) will sound largely indistinguishable from any other that can drive the same speaker without difficulty.  Blind, level matched testing seems to bear this out to a surprising degree.  

"Alan Shaw of Harbeth makes a similar point, although he is often misquoted as saying "all amps sound alike."  He actually claims that level-matched comparisons between properly functioning amps within their normal operating capacity will sound indistinguishable."

Tell him it is true for speakers, too. Take his blood pressure afterwards.

Ha Ha! Yes, imagine someone believing all speakers which measure the same sound the same.  There are intangible characteristics which are not necessarily measurable that make a difference.

smrex13  My experience in my large listening room with the Legacy Focus speakers (efficient with a low impedance load) includes an EAR 890 amp which cannot play dynamically or bass (yet does fine on Legacy Signature IIIs with an easier impedance maybe due to 3-10" instead of 3-12" woofers).  I currently use a pair of 125w. Class A/B monoblocks which is overkill.  However, I also have used a 35w. modified Dynaco Stereo 70 to great result (it is not ultralinear but voltage regulated design using a different tap off the transformer just like the monoblocks).  It has tremendous bass and dynamics.  It also has more neg. feedback and is less detailed than the monoblocks which is why it is used on the Signature IIIs.  So you're correct that the amp that can offer high current even at low power can really drive an efficient speaker with low impedance.  I've also tried a big Counterpoint hybrid amp (forgot which one, maybe the 200), a Bryston 4B and a pair of classic old Yamaha 200w. monoblocks.  The last three sounded polite (anemic) and lacking in tonal color compared to the two non-ultralinear tube amps.

I agree with almost everything @soix  said above. Great post!

As some as said, you need to define where in the price range of amps you think it doesn't matter anymore. Do you really not hear any worthwhile difference between a $100 amp and a $10 000 amp? 

OP might also want to read up why double blind tests are not always the best way to test audio.

Try isolating the amps sometime. It’s a whole different ballgame. Kind of the difference between mid fi and the high end. Even the guy off the street can hear the difference then.  All the dudes, like the bullet headed guy from Stereo Review, who made those grand pronouncements that amps all sound about the same predated proper vibration isolation, which didn’t become a cause celebe until the late 90s, a perfectly valid reason people don’t hear differences in cables, either.