At the top of the line is it really all that different?

You pose an excellent question, for which there is no right answer, everyone is different. I think as you go up the expense ladder, particularly with electronics, the qualitative differences are real, but narrow significantly. You may spend a good amount of money to get only a 5% gain but for some people that will be money well spent, others not so much. For some, the chase  is the game and is unending. 

This is also relative, not absolute. Someone who owns $50,000 speakers might think they are in  such rarified air that all that can be achieved has been achieved. The fellow who owns $100,000 speakers would think a little differently. 
 

in the final analysis, it’s not a competition, something we sometimes seem to forget.  All of us have a different budget and a different amount of money that we are able and willing to spend on this hobby. The goal should be to deploy those resources in a way that brings you the most enjoyment. Some will reach a place where they are perfectly satisfied and get off the train. For others, the journey doesn’t end and they always look for the next upgrade. Neither is wrong

There’s some truth in it.  I think there is a level of hifi gear that annoys you with some kind of sound or coloration, or for example bad bass or brash highs.  “Do no harm” applies first.  After that, to include finding the kind of sound you like and having a good room set up, the degrees of separation can be marginal.  
 

Unfortunately you can find yourself in the first phase (something’s not right) even after spending a lot of money.  Could be synergy between components, room, etc.  Having a good dealer helps.

@kerrybh +1

I’ve been contemplating how to give a thoughtful answer to the thoughtful question posed by the OP. I don’t think I’ll do better than the above but will add my $.02.

I’ve enjoyed every stage of my systems development over the years. As I made one improvement, I found myself desiring to make other improvements - because it was both fun to see (hear) “what if” and musically rewarding. It was also sending me down the proverbial rabbit hole. Upgrade here led to upgrade there. 

I have several times concluded I was done, just wanting to sit back and enjoy the music, only to eventually explore another what if. 

Where am I at now? Well, I stumbled into the category you are inquiring about with my DAC/preamp and cables. The speakers and amp are certainly nice, but fall well short of TOTL. Each piece was chosen based on a combination of performance, what my room would accommodate, aesthetics (I don’t want to look at gear), budget, and what was available from dealers I know, like and wanted to support.

Based on that criteria, in my room (a notable limitation), I find my speakers give me 95% of what I’ve heard from the best (other than MBL 101 MK IIs - but those didn’t meet the aesthetic requirement). They’re very satisfying. Will I ever upgrade? I can’t say.

On the DAC/preamp and cables? They’ve given me more of everything, allowing the speakers and amp to give their best. And yes, to me it was worth it though I can easily imagine others thinking me quite the fool. But they really were notably better than the gear they replaced (which was already good) and what I heard elsewhere. Better than other TOTL gear? Doubtful, but I had sound reasons for the choices I made.

Is it all worth it? I would never say one way or the other for anyone else other than to say never risk your financial independence over audio gear. Assuming that’s not a concern, I would suggest you step gently and try a piece here and there to see if it brings you the satisfaction you desire. It can be a fun journey, though be forewarned: I’m pretty sure no one actually gets to the final destination. 

What’s next for me? Not sure. I still wonder about my amp but substantively improving it would require a big spend - and I’m not sure I’d get that much more anyway. And, I am considering the Taiko extreme and Olympus servers - though that feels like it may add complexity I don’t need. And they are changing things quickly in that space. 

So today, I’ll enjoy the music. Hopefully I’ll do the same tomorrow.  And then I’ll go AXPONA and I may stumble again.

Best,

In my experience yes.  But I don't always need to be listening to a high end system to enjoy music.

Let me explain - I have experienced significant upgrades in my system over the years.  I notice them and enjoy them.  And I've listened to music at audio shows. in my local audio dealer and at folks houses.  I've had times where I listening to music was awesome and it was powered by a modest system.  I've had times where listening was 'ok, time to go' and the system was very high-end and put together.

I don't expect to upgrade again and I do enjoy listening to different music and different music.   

Enjoy  your journey!

We've been here before so many times. How well you can hear means a lot, just as how important this audiophilia nervosa is to you. I personally would not spend on audio too much even if I easily could, but it would still be tens of thousands dollars. Many components in the chain, so the cost is high for excellent sounding system.

However, after a certain level major limiting factors are recordings and room acoustics. Hard to change the former, right ?