At the top of the line is it really all that different?

At the higher end, parameters like resolution, clarity, soundfield nuances, bass impact, extension, etc can start to equalize, which are the same parameters that are immediately noticeable to a listener. Such equalization happens way more quickly with amplifiers, dacs, cables, etc. Subtle nuances don’t matter to most listeners, especially older ones with some hearing degradation.

At that point, you are better off improving the room and expanding your music collection. Or add a subwoofer, which can cause a drastic change...Or better yet, expand into object based multichannel audio/atmos, etc.

I believe it was Dali (could be wrong) that did a study where no one could distinguish speakers too much in a well treated room. If you insist on hearing differences that are wallet justifiable, get drastically different types of speaker designs. For example, i was comparing a TAD (concentric driver) and a Pioneer XY-3B (higher end PA horn that can bring down the house!) back to back in the same room for a bit. I could enjoy them for what they were... different presentations...could say the same with a recent Tekton

If i had been comparing the TAD with some higher end concentric driver  like a KEF blade, it could have been a "why bother" moment, if i didn’t have a room that is capable of putting everything under the magnifying glass.....There are so many similar 3 ways or 4 ways which may lead you to a "why bother" moment....magico,...schweikert...Rockport...etc, Meh....

It can be very rewarding to play at the margins using your idea.  I just did it with great results.  For example, the speakers I selected are NOS B&W 802 D Diamonds.  So they are both one step down from the top 801, but they are also 2nd generation vs. the current 4th generation.   Practical result:  $23K saved.  Soundwise, they are approximately 4 db down in the upper range, bass is little changed.  I, like many others actually prefer the Diamonds for home use.  Compared to the 801s, they are identical from 375 Hz up, but they are leaner sounding.  Most, including me, would prefer 801s given the choice.  Another example:  Amplifier.  I chose the recently discontinued Marantz PM10, an amplifier that the dealer said outperformed any other that he sold on B&W 802s.  He also sold Mark Levinson, McIntosh and so on.  Practical result:  At least $4k saved, the price of the PM10 before discontinuation.  It sounds terrific to me and my standard of reference is the C-J 27A class A amp I use in my main system.  There is no point in further examples.  If you are smart about it, there is no question that it is possible to put together a wonderful system at significant savings using your idea.  Most importantly, it is a lot of fun.

Decreasing marginal returns are as true in audio as anywhere else - possibly truer, considering how much one can spend with audio gear.

Value... different, and totally personal assessment. To me, $50k in audio is "unthinkable", but I have happily spent that amount or more on other stuff I collect. Part of the reason why it's 'unthinkable' is because of those marginal returns. I listened - briefly, and probably not attentively enough - to a DAC in a shop (Chord Hugo TT2) against its 4x-as-expensive brother (Chord Dave + MScaler). Yes, there were differences, but not worth spending $12k for me. Ended up taking the Hugo TT2 home for a couple of days, and returning as it wasn't "so much better" than my 30 year old LFD.

I'm quite happy with where my system is at, but like most of you, it's not a stable condition. I still look at gear every day, wondering if the $40k (used) speakers will be appreciably better than the $15k (used) that I spent. Same goes for the electronics. I'm tempted to go back 5-10 years to get premium used gear. 

You pose an excellent question, for which there is no right answer, everyone is different. I think as you go up the expense ladder, particularly with electronics, the qualitative differences are real, but narrow significantly. You may spend a good amount of money to get only a 5% gain but for some people that will be money well spent, others not so much. For some, the chase  is the game and is unending. 

This is also relative, not absolute. Someone who owns $50,000 speakers might think they are in  such rarified air that all that can be achieved has been achieved. The fellow who owns $100,000 speakers would think a little differently. 
 

in the final analysis, it’s not a competition, something we sometimes seem to forget.  All of us have a different budget and a different amount of money that we are able and willing to spend on this hobby. The goal should be to deploy those resources in a way that brings you the most enjoyment. Some will reach a place where they are perfectly satisfied and get off the train. For others, the journey doesn’t end and they always look for the next upgrade. Neither is wrong