The Audio Science Review (ASR) approach to reviewing wines.

This post was inspired by a question posed by @kjl1065 in a post in Tech Talk titled  'Seeking a Power Conditioner'.

In it the OP wrote:  "Read reviews on both Niagra 1200 & Puritan Audio PSM 156 power conditioners and the reviews were extremely positive. While reading I came accross a review of both products by Audio Science Review (ASR) who claims his reviews are objective with scientific data supplied and his reviews were not nearly as positive to say the least. Anybody have any thoughts on how I should proceed with the differing of opinions."

There was clearly an opportunity to voice an opinion about the ASR approach to audio reviews that in this case was based only on measurements but without any listening, and so the wine analogy was born and I thought it would be fun to expand on it here.

This was not meant just an exercise in ASR bashing, but as an exercise in educating those who might not be as familiar with ASR that they should be wary of their absolute opinions.

I believe that objective measurements are important, but so it the subjective listening experience.  Objective and subjective balance and complement each other as we see in the Stereophile approach to in-depth reviews.

Interesting discussion.  I read ASR for a while because I found the reviews with data helpful. I also read and listened to reviews on other sites that discussed listening impressions.  Put them all together and you get a sense if a piece of equipment is worth checking out.  The thing that turned me off about ASR were some of the comments that were posted.  If someone posted they liked a piece of equipment I would often see posts using measurements to challenge the post.  Then I would see it get personal with comments back and forth that were very negative.  I listen to music and read about equipment as a hobby to get away from the day to day grind and all the negativity that seems to be at an all time high.  Life is too short.  I don’t have time for negativity.  One of the things I like about this site is I see that people share their opinions and they’re respectful and usually positive.  As Hans says, enjoy the music.  

@analog_aficionado Outstanding post (the long one). Helps very much to understand and reconcile the differences between ASR results & listening. Thanks!

I am in the camp that says ASR is a single source of information just like every other review. 

measurements are critical to audio, but a 100 grams of german chocolate measures the same 100g as  100g of dog crap. they dont taste the same.

a coupla other observations:
refusing to grasp the limitations of measurement is some form of flat-eartherism.
"objectivists" are ayn rand devotees. that has nothing to do with hifi audio.

As for the amplifiers that I’ve owned or borrowed and enjoyed the most, none really offered what anyone would refer to as spectacular measurements.

"Measurements" is too crude a word. @prof was pointing at this issue and this comment ignores it.

Some measurements are, say, 2nd harmonic distortions -- those may upset some at ASR, but the rest of us understand that those measurements are NOT aligned with "bad sound" as some of us experience that. (Others here do NOT like that 2nd harmonic. So, this varies.)

But other measurements are of a kind that correlates to what we would ALL agree are responsible for a bad-sounding product. Some of us here applaud ASR and others for measuring things which DO correlate with "bad sound."

As @prof put it:

It’s like so many audiophiles imagined that measurements are just plucked out of the air for no reason at all. The whole point of measurements is that they have been correlated to how things sound. That’s the point of measurements! And scientific study has shown that certain measurements correlate to what most people will rate as higher quality.

Of course, there are issues with how ASR folks do things, as @analog_aficionado points out:

Objective measurements are great tools insofar as the results are understood and interpreted properly. This is where the current debate seems to run into trouble. Take SINAD (aka THD+N) for example. There seems to be a monomaniacal over-emphasis on this metric as an end-all-be-all measurement which somehow dominates the subjective performance of a piece of equipment over most other aspects of performance....I would even argue that THD in the context of electronics is increasingly irrelevant, given how low distortion is in most modern designs. Turn to another famous objectivist like Ethan Winer, and you’ll find excellent demonstrations of the audibility of THD.

So why chase after 0.0002% THD in a DAC or amplifier? I’ve built, lived with and loved tube amplifiers with rather embarrassing distortion figures compared to the modern benchmark....So measurements have their sensible limits as well. It does no good to go overboard with a single specification.

It’s a complex debate. My main issue is how people simplify the issues too much. Maybe people like to remain vague on what a "measurement" is because they like to "take a stand" against so-called "objectivists" or "measurementalists." But that is not playing fair with language and the result is to perpetuate misunderstanding.

Finally, I am certain that @audio_aficinado nails it with this comment:

But I do know this: a ruler (even a really really awesome one) is just not good enough. The human perception of sound is not well understood and even less well quantified, and there are many aspects of objective technical performance of audio equipment that we already know something about which are being overlooked.

At the end of the day, it all boils down to this for me:

Objective audio measurements must by definition be subservient to the Subjective outcome. If not, then we aren’t talking about hi-fi anymore.

I had a long and bitter debate with Ethan Winer about this. My position is that there may be things heard which cannot be measured because the brain and perception are way WAY beyond our understanding at this point in our scientific understanding. His reply to that was, essentially, "No, it’s just placebo effect and subjective bias." (In other words, y’all are just in denial.) He could be right about that claim, but he has no basis for making it, and we’re just back in the realm of rhetoric, not argument.