Great topic...
During the past few years I have reducing my collection, replacing gear, rolling tubes, etc. So I have had the same challenges, thousands of times.
To me:
A. 'critical listening skills' are just that... critical. When needed, I listen to the attack of a bowed violin, the fingering of a oboe, the plucking Fender bass, or cymbal crass. I listen accuracy of tone and harmonic structure. I listen to decay (is it natural, truncated, over-extended?) I also listen for micro-dynamics (did a flute-players air wobble, or did a tape-splice change the volume of a acoustic guitar?).
Yet, critical listening is for a comparison, and should never be a life-style.
B. the other-side of the coin is the wholistic or gestalt experience eloquently discussed above by so many...
I prefer many days of hearing music on new gear (especially speakers), allowing myself to remember this music played on my earlier gear... the result is very satisfying. I know my efforts have paid-off when I relax and truly hear the music.
In other words, I am no longer searching for a sound, I can simply let the music come to me...
In my home, critical listening and casual hearing both play a vital role.
I hope this is fertile ground...