Great topic for a thread, I've dealt with this problem all my life. When I bought my first system as a teen, my goal was to build a system that allowed me to hear music at it's best.
Like you, I found that often with high end components, the sound was not satisfactory unless "audiophile" recordings were used. I hate most of those records, our group refers to them as terrific recordings of bad music :^).
It's taken me many years, but I have my system where I can play every kind of music and the sound is good. The really good stuff is beyond compare and the bad stuff is tolerable and completely enjoyable.
I think that's the best that can be achieved. The fact that a high end system can be built to play all music is a testament to the options available to us. There are many ways to get to this point, including great but forgiving speakers and great but forgiving electronics, or both.
The key is getting ultra low distortion and ultra low noise. With that as a starting point, any distortion in the recording becomes a separate entity, and something that identifies "that" recording from the others. I have LP's that are 50 years old that sound better than recordings made last year.
If a system is built just right, you can get perfect sound from perfect recordings, good sound from good recordings and acceptable (even fun) sound from poor recordings.
In my case (and for my taste), the key is first class analog, powerful tube components and extremely low distortion speakers. There are things about that combination that are pleasant to the human ear and leave headroom for bad recordings of great music.
This last Friday night our group listened for over eight hours and no audiophile LP's hit the turntable. The majority of that nights music was the blues, with much being from 1940's through 1968. We had a wonderful time and I'm still getting email from members saying how great the sound was. Oddly enough, this was a "cleansing session" from the incredible quantity of drum records, Patricia Barber and similar software that populated the rooms at RMAF.
The answer to your question is yes, it's possible to get great sound without sacrificing the quality wonderfully recorded music.