At the top of the line is it really all that different?

It depends on you. Are you thrilled by the incremental difference? I am. It is what has been so rewarding about this pursuit. That additional realism and the fit with my aesthetic... which is now how well duplicates real natural music. Both aspects make me extremely happy. So, that appreciation brought me to identify Wilson, Magico, B&W, and Sonus Faber... but to purchase and enjoy Sonus Faber... because of its excellence and its fit with my aesthetic.

If I was a "good enough" kind of person... well it would be a complete waste of money. It is all about what you get out of it and that is a personality trait. This is typical of people across the spectrum. I hike, climb, and bike ride. I have a closet of Arctrex jackets that are both in themselves the pinnacle of design excellence and with the selection I have I can go out into sever conditions and be perfectly comfortable, with the perfect garment. It increases the enjoyment to me of the experience. Ditto... cameras, computers... etc.

There are times I find that being as poor as I am, that I am still terrified that my investment is not going to do any good. Then to find that it completely blew away my expectations. Hopefully the day that this doesn’t make a difference is a long way away for me. 

At the higher end, parameters like resolution, clarity, soundfield nuances, bass impact, extension, etc can start to equalize, which are the same parameters that are immediately noticeable to a listener. Such equalization happens way more quickly with amplifiers, dacs, cables, etc. Subtle nuances don’t matter to most listeners, especially older ones with some hearing degradation.

At that point, you are better off improving the room and expanding your music collection. Or add a subwoofer, which can cause a drastic change...Or better yet, expand into object based multichannel audio/atmos, etc.

I believe it was Dali (could be wrong) that did a study where no one could distinguish speakers too much in a well treated room. If you insist on hearing differences that are wallet justifiable, get drastically different types of speaker designs. For example, i was comparing a TAD (concentric driver) and a Pioneer XY-3B (higher end PA horn that can bring down the house!) back to back in the same room for a bit. I could enjoy them for what they were... different presentations...could say the same with a recent Tekton

If i had been comparing the TAD with some higher end concentric driver  like a KEF blade, it could have been a "why bother" moment, if i didn’t have a room that is capable of putting everything under the magnifying glass.....There are so many similar 3 ways or 4 ways which may lead you to a "why bother" moment....magico,...schweikert...Rockport...etc, Meh....

It can be very rewarding to play at the margins using your idea.  I just did it with great results.  For example, the speakers I selected are NOS B&W 802 D Diamonds.  So they are both one step down from the top 801, but they are also 2nd generation vs. the current 4th generation.   Practical result:  $23K saved.  Soundwise, they are approximately 4 db down in the upper range, bass is little changed.  I, like many others actually prefer the Diamonds for home use.  Compared to the 801s, they are identical from 375 Hz up, but they are leaner sounding.  Most, including me, would prefer 801s given the choice.  Another example:  Amplifier.  I chose the recently discontinued Marantz PM10, an amplifier that the dealer said outperformed any other that he sold on B&W 802s.  He also sold Mark Levinson, McIntosh and so on.  Practical result:  At least $4k saved, the price of the PM10 before discontinuation.  It sounds terrific to me and my standard of reference is the C-J 27A class A amp I use in my main system.  There is no point in further examples.  If you are smart about it, there is no question that it is possible to put together a wonderful system at significant savings using your idea.  Most importantly, it is a lot of fun.

Decreasing marginal returns are as true in audio as anywhere else - possibly truer, considering how much one can spend with audio gear.

Value... different, and totally personal assessment. To me, $50k in audio is "unthinkable", but I have happily spent that amount or more on other stuff I collect. Part of the reason why it's 'unthinkable' is because of those marginal returns. I listened - briefly, and probably not attentively enough - to a DAC in a shop (Chord Hugo TT2) against its 4x-as-expensive brother (Chord Dave + MScaler). Yes, there were differences, but not worth spending $12k for me. Ended up taking the Hugo TT2 home for a couple of days, and returning as it wasn't "so much better" than my 30 year old LFD.