I have friends that are audiophiles more interested in "sound" than they are music. For them, music is a means to an end, the end being improvements in the quality of the sound.
I am more interested in the music. I am an audiophile as a means to the end, the end being getting closer to the music. There are a lot of audiophile "qualities" of the sound that make the music more enjoyable, but the end-game is the music, not the sound.
So my suggestion is to find the equipment that meets your needs and don’t worry about scaling the mountain path of constant upgrades. If it’s sound, you will never reach the top. If it’s music, find what gives you the most enjoyment and enjoy the music.