How can you tell the quality of the recording?

Let me interject a little logic here. IF, your system HAS EVER played a recordings that met all the positive things you are seeking in reproduced sound, even JUST ONCE, that means that the system is capable of playing at the highest levels. So if the next recording does not sound up to expectations, IT HAS to be the recording. Unless your system has a grudge against Jazz, or classical or tom waits etc....... then it's personal.

My system doesn't like country music. It always sounds bad when someone makes me play it.

the hair on my neck does a pretty good job of figuring out how good a recording is. amount of teeth grinding comes in a close second.

The biggest flaw is treble harshness. Other problems can include boominess, especially in the bass, and timbral distortion that makes the instruments or voices come across as fundamentally different from live sound.

Problems with soundstaging are very secondary.

I disagree with Rok2id. I'm sure there are systems out there that only sound really good with just one recording. That does not mean the problem is with the other recordings.