I think one form of critical listening just means dedicated listening - you're listening to music as your primary and only activity, usually in an environment that is set up to provide the best sound - ie, fully utilize your system. Or, perhaps, this is better known as "serious" listening. I like Bomarc's concept of "just listening" as it sounds a lot more light-hearted.
I think there is another form, often used by reviewers and, hence, re-used by audiophiles, of critical listening meaning that you're looking for strengths and flaws, and trying to determine how to maximize the former and minimize the latter. This is more listening to gear than listening to music and, as you suggest, is aking to trying to find nits on a bald head. It undoubtedly detracts from the experience of listening to music because it changes the whole scope and focus. Many reviewers, in an attempt to show their excitement over a particular component while evaluating it through critical listening, admit that they just "got lost in the music" - ie, they started "just listening". Whoops!