I think Audiophilia attracts certain types of people. Men for one, but with analytical skills, a love of nuance and subtlety, lots of cuiousity, and definitely somewhat obsessed. Depending on where you are on the scale of obsessed will probably determine if you get dragged too far.
For me, the reward always outweighed any other aspect. I have been obsessed for 50 years nonstopped. I enjoyed listening to my system (sometimes the music, sometimes the system). Although at one point (about fifteen years ago) I had pushed my system too far to the detailed / analytical side. I loved listening to it for 45 minutes… I was working, so that was all I could manage most of the time. If I had more time I would get bored and go do something else. I was disappointed… and wondering if my interest was waning.
I called this system my reference system, as I could immediately determine the nuances of the venue and mastering techniques. It seemed like the best possible system.
Then I upgraded my headphone system to an unbelievably good, really powerful 300B amp. Instantly what was wrong with my main system became obvious. It lacked warmth and musicality as well as rhythm and pace. I completely rebuilt my main system… it is now completely musical and enthralling. I listen three hours a day and have to be dragged away each day.
Sounds like for you it was a change in attitude, for me it was a change in the sound of my system.