There is an assumption on the part of the OP and others that audiophiles spend more money based mainly on sound quality. I would challenge this basic assumption.
This hobby has morphed into a luxury pursuit where there is no longer any practical limit on the cost of gear. There are speakers, for example, that retail for over $1 million. Burmester released mono amps that cost $350,000 a pair. I could go on. At the same time there are dozens of documented blind tests that show that listeners can't tell most gear apart on sound quality. Therefore, I think the question of diminishing returns is mostly irrelevant.
We buy and upgrade our gear because it makes us feel good to do so. We get an emotional response to the story behind the product, the design and aesthetics, input from members of our tribe, and the brand identity. Confirmation bias will insure that it sounds better than the previous stuff and there's nothing wrong with that. If it makes us happy then it was money well spent.
In my direct experience I have tried to determine if upgrading will bring a worthwhile benefit. The core of my system is about 25 years old (Thiel CS6, Krell KSA300s, etc.). I could afford to buy some fairly expensive new stuff if I really wanted it. I've been to 3 audio shows and I have heard a bunch of 6 figure systems that didn't sound better than my setup and some that sounded considerably worse. I have also heard a few systems that I would love to audition in my home; MBL 101 E Mk II, for example. But if I spend the money it will be as much for the pride of ownership as the improvement in sound quality. I would simply love to have a pair of 101 E's plus a pair of those beautiful black MBL monoblocks in my listening room. You can bet they would sound glorious.