At the top of the line is it really all that different?

It can be very rewarding to play at the margins using your idea.  I just did it with great results.  For example, the speakers I selected are NOS B&W 802 D Diamonds.  So they are both one step down from the top 801, but they are also 2nd generation vs. the current 4th generation.   Practical result:  $23K saved.  Soundwise, they are approximately 4 db down in the upper range, bass is little changed.  I, like many others actually prefer the Diamonds for home use.  Compared to the 801s, they are identical from 375 Hz up, but they are leaner sounding.  Most, including me, would prefer 801s given the choice.  Another example:  Amplifier.  I chose the recently discontinued Marantz PM10, an amplifier that the dealer said outperformed any other that he sold on B&W 802s.  He also sold Mark Levinson, McIntosh and so on.  Practical result:  At least $4k saved, the price of the PM10 before discontinuation.  It sounds terrific to me and my standard of reference is the C-J 27A class A amp I use in my main system.  There is no point in further examples.  If you are smart about it, there is no question that it is possible to put together a wonderful system at significant savings using your idea.  Most importantly, it is a lot of fun.

Decreasing marginal returns are as true in audio as anywhere else - possibly truer, considering how much one can spend with audio gear.

Value... different, and totally personal assessment. To me, $50k in audio is "unthinkable", but I have happily spent that amount or more on other stuff I collect. Part of the reason why it's 'unthinkable' is because of those marginal returns. I listened - briefly, and probably not attentively enough - to a DAC in a shop (Chord Hugo TT2) against its 4x-as-expensive brother (Chord Dave + MScaler). Yes, there were differences, but not worth spending $12k for me. Ended up taking the Hugo TT2 home for a couple of days, and returning as it wasn't "so much better" than my 30 year old LFD.

I'm quite happy with where my system is at, but like most of you, it's not a stable condition. I still look at gear every day, wondering if the $40k (used) speakers will be appreciably better than the $15k (used) that I spent. Same goes for the electronics. I'm tempted to go back 5-10 years to get premium used gear. 

You pose an excellent question, for which there is no right answer, everyone is different. I think as you go up the expense ladder, particularly with electronics, the qualitative differences are real, but narrow significantly. You may spend a good amount of money to get only a 5% gain but for some people that will be money well spent, others not so much. For some, the chase  is the game and is unending. 

This is also relative, not absolute. Someone who owns $50,000 speakers might think they are in  such rarified air that all that can be achieved has been achieved. The fellow who owns $100,000 speakers would think a little differently. 
 

in the final analysis, it’s not a competition, something we sometimes seem to forget.  All of us have a different budget and a different amount of money that we are able and willing to spend on this hobby. The goal should be to deploy those resources in a way that brings you the most enjoyment. Some will reach a place where they are perfectly satisfied and get off the train. For others, the journey doesn’t end and they always look for the next upgrade. Neither is wrong

There’s some truth in it.  I think there is a level of hifi gear that annoys you with some kind of sound or coloration, or for example bad bass or brash highs.  “Do no harm” applies first.  After that, to include finding the kind of sound you like and having a good room set up, the degrees of separation can be marginal.  
 

Unfortunately you can find yourself in the first phase (something’s not right) even after spending a lot of money.  Could be synergy between components, room, etc.  Having a good dealer helps.