Affordable vs. ultra expensive speakers - what's the difference?

It’s difficult to judge a pair of speakers based on the frequency response plot

True, that's why I look at more than the FR plot. If I can't find extensive measurements on a speaker I don't even look at it. 

Nobody has mentioned the true difference - your ears. 
In a true double blind test scenario, nobody here could tell you what speaker they were listening to - no matter the price. 
 

buy what sounds best to you- measurements do not mean a thing if you can’t hear the differences - then simply enjoy. 
 

The difference = a lot of money for about 2-5% improvement in sound.....

The curve of diminishing returns.

Will it sound $48,000 better?  No

Will it sound better than a $2,000 speaker? Yes

For the wealthy people, $50k means nothing. For the average joe, there are many other ways to spend that money for good usage. 

Sound systems need to have synergy with their components. A great sounding system could cost less than 3k in the used market.

Not everything in life can be measured or ranked in purely monetary terms.

Pride in ownership and the daily pleasure that an object may bring one applies to Audio equipment as much as it does to anything else, and as such cannot be measured by anyone other than the individual.