What do you say to your audiophile friends who say you just listen to your equipment more

It's always been about the music. The audio gear is just a way to get there in audibly satisfactory way.

I tell them that they’re right (sometimes) and also point out that they are smarter than I because they can be satisfied with a $50 Bluetooth speaker while I have to spend at least 20% of my income to truly enjoy the sound... 

Great sound enhances the appreciation of great music.  The gear is a means to an end, not an end in itself

I think I'm a music lover first and I think we all are looking for the magic sound!! The gear is the vessel to get us there!!

It's all in service to the music.
A handheld transistor radio (kids, look it up) can sound like a symphony if it's all you've got.
However, if you're a gear-head just to be a gear-head, who cares....own it and let the devil take the hindquarter (as my grandmother used to tell me)