OP, I think you’ve said that -- for you -- "relaxing" means not thinking critically or doing analysis.
Sometimes, that’s true for me. But I enjoy critical thinking, too. So, sometimes, relaxing for me involves critical listening. Or doing a crossword puzzle.
I don’t see different "camps" in this question, as some do. I see mere differences in how people relax, and not much more.
Now, if people have trouble shutting off their critical listening and are unable to relax and enjoy the music, this is saying nothing more than they're unable to relax.
And that is not really about being an audiophile. It's about their psychology in general.