In the end, it depends on how much obsessiveness and pathological compulsion dedication and commitment one is willing to bring to the table.
As @onhwy61 rightly pointed out, a purpose-built listening room becomes a necessity if one is to even hear the difference better gear makes. Yes that room can cost $250K+, though it doesn't have to.
I don't have a dedicated listening room and I don't want one. I sometimes listen to music 12 hours a day so I need to do something else at the same time, namely, in my case, work. That supposes, by definition, a very imperfect room, which in turn limits how much better gear I can get while still reasonably hoping to hear significant improvement. But I wouldn't last an hour a day in a dedicated listening room before I got cabin fever, so I'm more than willing to live with what I have as it lets me listen to music for another 11 hours, and at the end of the day listening to music is what I'm here for.