What s Your Frame of Reference?

Whenever I make a change to my system I pull out a select few records to evaluate the "upgrade". Gross differences in sound quality are fairly easily judged, but most likely the change in sound quality is subtle and better judged over an extended listening period. This is my problem: let's say I change something and on one of my reference records the trumpet now sounds a little more brash and upfront, maybe even bordering on harsh. How do I know whether the upgraded system is more accurately portraying the sound of the recorded trumpet, or has the upgrade merely added an upper midrange resonance problem? I have a good idea of what a generic trumpet (w/ and w/o mute) sounds like, but I wasn't at the recording session. The studio, the mic, EQ, recording medium, etc. all add an enormous amount of variables to what is actually recorded onto the record. If I judge the sound to be harsh and make changes to my system to remove the harshness, then maybe all I've done is make the system more pleasant, euphonic, but less accurate. The "live music in real space" paradigm is not particularly useful in that the overwhelming majority of the music I listen to is not of this type. Besides, the transparency of the audio engineering is still a variable. Ideally, I need a wide bandwith recording where I was present at the recording and which the engineer faithfully recorded the music. Unfortunately, I don't have such a recording. How are other dealing with this issue?
Very true, Onhwy61. I use recordings, such as Chesky's "Ultimate Demonstration Disk" that give an introductory explanation of what & how we're supposed to hear the track that follows. Apart from the repetitive boredom, it can serve as a benchmark.
But then, as you say, nor was I at the recording session (or the mastering...)...
I wonder what others have to say?
An interesting question. Gregm's suggestion of the Chesky disc is a good one. The Stereophile recordings have detailed notes as to what they think the recording sounds like, those are also helpful. Otherwise, I try to play well-recorded recordings, and a good number of them, from a cross-section of labels (Reference/Harmonia Mundi/Telarc/Delos/Decca, for example), to see if the component in question is consistent in its reproduction of the same instrument/vocal range, etc. from recording to recording and label to label. Not an exact science, of course. Some of the Delos recordings are of performances of the NJ Symphony I attended in the NJ Performing Arts Center, those are a good reference for me.
I play a broad selection of CD's (some of them not being quality recordings at all). I focus on specific instruments (and even notes) but also on how the music sounds as a whole. I prefer that our system sound good on a variety of material instead of just sounding great on great recordings and not so great on run of the mill productions. Like I have said before "if Mitch Ryder doesn't sound half way decent, then it's not for me".
Hi hwy61; Are you trying to "unscrute the unscrutable"? I also do as Rcprince and Dekay do, ie play a broad variety of music that I am especially familiar with. For me, it's the timbre and character of vocals-- both male and female that are my references. If vocals don't sound natural, It's been my experience that instruments don't sound right either, eg piano and acoustic guitar.

I like Holly Cole, Margo Timmins, Diana Krall, Shirley Horn, Melissa Ethridge, JJ Cale, J. Cash, Buddy Guy, Koko Taylor and others as references-- real as opposed to "reference recordings". And I reserve the right to decide what is "natural". BTW, I've played both acoustic and electric guitars for 40 years, and am very familiar with what they sound like, including guitar amps.

But then, I don't think I'm after absolute accuracy. Like Dekay, I want my system to sound good with a broad variety of music. So I'm always asking "does this sound natural"? rather than "does this sound accurate"? I may use 10-20 well recorded CDs to make decisions regarding system changes. As I can never know what the original recording actually sounded like, I'm content to go with what sounds "natural" to me, even though it may not be closest to absolute "accuracy". This may just be semantics though? Craig.
I do the use the Chesky disc. It's very good for it's intended purpose. In practice I following what Garfish does, but I wonder if I could be boxing myself in. My background is that I've assembled a system that gets the basics done. Obviously, it doesn't sound exactly like live music, but it's very high quality reproduced sound. It's comparable to what you would hear at a well done dealer demo of high quality equipment. Any changes to the system I would now make are fine tuning, but I wonder if I'm just making my record collection (both vinyl and CD) sound nice, as opposed to getting at what is actually on the records. Part of what got me thinking of this is my recent acquisition of a 5-band, parametric equalizer which affords me an enormous ability to vary the tonal quality of the music. Maybe I'm getting too hung-up on this accuracy thing.

Thanks for the replies. Rcprince, I envy your situation.
as several others have already mentioned, you ought to play a wide array of recordings to judge how a component changes (or doesn't change) the sound of your system. i think it's most helpful to have in your "test bin" several recordings you've heard on your system as well as a number of other systems, be they your friends', dealer's or those you encounter at audio shows and the like. i find test discs, such as those put out by ortofon, chesky, stereophile, etc. to be helpful in only the most rudimentary ways. they can tell you, for example, whether you've plugged in all your left/right ic's correctly or whether the cartridge azimuth is optimal; they are rarely helpful, however, at least to me, in determining the sonic "character" your system imparts.

as a real world example, i offer this: yesterday, i had a fellow 'phile and his wife come to my home to hear my turntable/tone arm setup to decide whether to purchase it. these are among the lp's we played to "test" the sound quality of my analogue setup: csn&y, "deja vu"; thomas dolby, "aliens ate my buick"; gary karr, "adagio d' albioni"; laurie anderson, "mister heartbreak"; paul simon, "there goes rhymin' simon"; prokofieff, "lt. kije; pink floyd, "the final cut." these were each recordings very familiar to both my potential buyer and me. we could have picked , perhaps, 20 or 30 more that would fit this description. the point is: we had "points of reference" as to these recordings. neither of us could judge if any one of them was more accurate on my system than that of "mr. x," but we could judge where the differences lay and which of those differences we preferred. that, i think,
somehow my above post got cut off. it should end with "...that, i think, is what comprizes a 'point of reference.' " -kelly
This may seem simplistic but I use a familiar recording to get a quick sense of the "tone" of the component, then the imaging. If these are favorable, I listen more carefully to try to discover something I haven't heard before in the recording. That's it. If it sounds "musical", images well and is more revealing, I have to assume it's a winner.
That assumption is the problem. Unfortunately, it's only in long term listening that you truly discover whether the CIQ (component in question) is right for you. But then for those of us drawn to Audiogon, selling it is half the fun.
I am lucky enough to have a number of recordings that I have made, and can judge a new component that way if I have to, but I usually do something similar to Garfish and Cornfedboy. Also, I have heard my favorite tracks on 3 different state of the art systems, and am very familiar with the sound of one of them, so I use that as a reference to compare my system to. Now that I know exactly how close my system is to that reference,
I can tell if any changes are taking me closer to that ideal, or not.
I just pick a disc that I know like backwards. Much easier to discern any differences that way (usually quicker anyways)
I also use several discs but mostly I go for symphonic music
to hear the violins and other strings.I consider this the ultimate test.Just think of the idea that there are probably about 60-80 musicians playing and you are hearing them through two boxes in your living room.What a challenge to make it sound good.
After the strings I go for piano music which is also difficult to reproduce.This is a good instrument to listen for decay-how long you hear the note after the key has been struck.
If an upgrade passes these two tests it is usually good on other types of music.
Kitch29 brought up a good point that I also agree with. The change (hopefully improvement) has to stand the "test of time", eg for me it has to be non-fatiguing over time. Craig
I like to use recordings of the human voice when making critical decisions. If they sound natural, then I feel like I’m on the right path. I second the recommendation of Johnny Cash mentioned above. His recordings from the past few years are very well done. Try “American Recordings.” Just Johnny and a guitar that is downright spooky in its realism. Then again, maybe there is some synergy between the way these are recorded and the sound of my system. I think this is THE great variable in evaluating audio gear. A reviewer in the mainstream audio press tells us that a certain recording has a certain quality. Such as “the most natural female vocals I’ve ever heard.” Of course this evaluation is based on the systems that this particular reviewer uses for his reference. If I choose to use this recording as my reference for female vocals, then I am going to end up with a system that has a similar tonal balance to that of the reviewer. Perhaps not a bad thing, but I’ll never really know. This paradox can drive me crazy. I think I’ll just go down to the pub and listen to the house band.
Depends what I am auditioning. I'll pick a big orchestra piece for trying an amplifier, but will go for Chamber Music trying cables. For a preamp I may pick a solo piano recording. If the piano does not sounds right, I can stop right there.
Sugarbrie; an interesting testing method(s)-- I really don't think I could call it that close. Cheers. Craig
First don't use a single CD or record for your reference. As your system gets better every CD will sound different. Some better and some worse so in auditioning new audio gear listen to a lot of music. Unfortunately sometimes you don't hear all the good and bad until you live with the sound a while. I've heard systems that sound "better" then my current system and brought a component home to fine the "better" also add a lot of listening fatigue. Be careful and get parts home to listen to them one component at a time.