Is there actually a difference?

Ha Ha! Yes, imagine someone believing all speakers which measure the same sound the same.  There are intangible characteristics which are not necessarily measurable that make a difference.

smrex13  My experience in my large listening room with the Legacy Focus speakers (efficient with a low impedance load) includes an EAR 890 amp which cannot play dynamically or bass (yet does fine on Legacy Signature IIIs with an easier impedance maybe due to 3-10" instead of 3-12" woofers).  I currently use a pair of 125w. Class A/B monoblocks which is overkill.  However, I also have used a 35w. modified Dynaco Stereo 70 to great result (it is not ultralinear but voltage regulated design using a different tap off the transformer just like the monoblocks).  It has tremendous bass and dynamics.  It also has more neg. feedback and is less detailed than the monoblocks which is why it is used on the Signature IIIs.  So you're correct that the amp that can offer high current even at low power can really drive an efficient speaker with low impedance.  I've also tried a big Counterpoint hybrid amp (forgot which one, maybe the 200), a Bryston 4B and a pair of classic old Yamaha 200w. monoblocks.  The last three sounded polite (anemic) and lacking in tonal color compared to the two non-ultralinear tube amps.

I agree with almost everything @soix  said above. Great post!

As some as said, you need to define where in the price range of amps you think it doesn't matter anymore. Do you really not hear any worthwhile difference between a $100 amp and a $10 000 amp? 

OP might also want to read up why double blind tests are not always the best way to test audio.

Try isolating the amps sometime. It’s a whole different ballgame. Kind of the difference between mid fi and the high end. Even the guy off the street can hear the difference then.  All the dudes, like the bullet headed guy from Stereo Review, who made those grand pronouncements that amps all sound about the same predated proper vibration isolation, which didn’t become a cause celebe until the late 90s, a perfectly valid reason people don’t hear differences in cables, either.

Back in the 1970s, so many receivers from major companies sounded the same-distorted.  They were designed around measurements, how low can the THD and IM go, not around musical sounds.  I dreaded hearing my teen age friends playing rock.  It hurt my ears/head.  I would go home and enjoy my Yamaha integrated or Fisher 500c into a pair of Dynaco 35s, Dual 1209 and Grado cartridge.  Cheapest cabling, no vibration isolation, no acoustic treatments (nice big room 20X20X10).  That was comforting sound.

Big, big differences between my Power Amps. Have 5 now, did have 6 until recently, but sold one. Why? Because it sounded inferior to the other 5. Easily heard.