Affordable vs. ultra expensive speakers - what's the difference?

The only thing for sure is a lot money (the difference). The rest all depends.

It’s difficult to judge a pair of speakers based on the frequency response plot.  There are just too many parameters.

It’s difficult to judge a pair of speakers based on the frequency response plot

True, that's why I look at more than the FR plot. If I can't find extensive measurements on a speaker I don't even look at it. 

Nobody has mentioned the true difference - your ears. 
In a true double blind test scenario, nobody here could tell you what speaker they were listening to - no matter the price. 
 

buy what sounds best to you- measurements do not mean a thing if you can’t hear the differences - then simply enjoy. 
 

The difference = a lot of money for about 2-5% improvement in sound.....

The curve of diminishing returns.