Factor 1 - Audiophile equipment ranges from relatively affordable to stuff that's very few could even consider as a possibility, so there's always something bigger and better available.
Factor 2 - Every audiophile has different references and unique listening spaces, so the only way to really know is by trying gear in your own listening space.
Factor 3 - Many audiophiles are in it for the hobby as much as for the music so upgrading and trying different gear is part of the fun.
Factor 4 - In my opinion, most people struggle to separate something sounding "different" with it sounding "better" unquestionable better.
At this point my system is reasonably on the audiophile scale of possibilities, but exceeds anything that I ever had as a goal, so I'm satisfied...for now.