Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

@cleeds 

I really don't think ASR has much power or influence at all. It's just a noisy group with grievances, which is very common today.

Well, I know more than one company that has completely retooled their entire product development and product strategy precisely because of the work we have been doing (think Schiit).  So we do have influence, having nearly 3 million visitors a month which puts us at or at the top of all audio based sites on the Internet.

The main grievances, as evidenced by threads like this, are people who don't want to come along.  They rather cling to "beliefs" than accept reality of objective and science based evaluation of audio.  Why?  Because in some cases it negates their praise for some product.  Instead of learning from that, they choose to go after ASR.  And for what?  Us providing more data than you had before about your purchases?  There is no better definition of asking, heck demanding, to put one's head in the sand.

By all means, do that but don't make up stuff about ASR.  At least do some homework before creating FUD like this.

amir_asr

.... you want to live in a fantasy world where differences that don't exist or are below audible, are so obvious that your wife can hear from the kitchen ...

You live in a fantasy world. You don't know anything at all about me. But you have an active imagination, I'll give you that.

@normb 

Basic measurements are only a benchmark, an objective standard, but how something SOUNDS is purely subjective and has to take into account intangibles like combined elements in the system, the room acoustics, speaker placement, and the listener, right?

Yes and no.  In most cases, audio gear is designed to have its own performance or you would never be able to assemble any system.  Or trust any reviewer whatsoever, right?  For example, a source device such as a DAC has a low impedance of say 100 to 200 Ohm.  The pre-amp then has an impedance of at least 10X that.  This way you get full voltage transfer which is what we want.

On the other hand, there are tube amps with high output impedance which then interact with the frequency response of the speaker.  This causes tonality to shift in the system not because of anything good, but because of poor design.  A solid state amp will have well below 1 ohm impedance as to eliminate this effect.  You could hear these effects using either listening tests (if the person is properly trained and difference large enough), or measurements.

In vast majority of cases though, the modular aspect of audio allows us to independently test and evaluate a component by itself.  Measurements are much more powerful in this regard because audiophiles as a group are terrible at detecting non-linear artifacts. But even for things like speakers where distortions are apparent, we are a) mostly alike when it comes to preferences and b) non-professionally trained listeners including reviewers and dealers are terrible at providing consistent and proper feedback.  Please see this formal study: 

Indeed, the subjective data from audio reviewers is so bad that you need 10 times as many of them to equal properly and formally trained speaker listeners!  That is the problem with subjective remarks from audiophiles or audiophile press.  It is so unreliable that it is not worth paying attention to.  The same study by the way shows that listener preference is similar among a dozen different listener classes:

See how the ranking of each speaker did not matter (different colors) regardless of who was listening to it (X axis).  Green speaker for example was bad no matter who was evaluating it, in controlled tests that is.

If you want to see a more detailed explanation of that, I have done a video on it:

No matter what you believe or what side of the argument you are on, I am quite sure that if all involved stuck purely to facts and what is being discussed, the conversation would be more cordial. Unsubstantiated personal slights, unsubstantiated accusations, and generally nastiness that has nothing to do with audio are not going to resolve anything.

@1extreme 

I used to follow their rankings because I thought well maybe at least they would identify a component that is a disaster but no longer. Why, because a few months ago I started listening again to my MHDT Orchid Tube DAC which as you probably all know is an R2R ladder DAC and I was amazed at the quality of music it was producing over my delta sigma DAC’s, especially with acoustical music like traditional jazz which I mostly listen too. Out of curiosity I looked on the ASR site and Amir had ranked the MHDT Pagoda DAC, which is the same as the Orchid DAC but with XLR outputs as the WORST DAC THEY HAVE EVER MEASURED. They have the Pagoda DAC at the farthest right in the red scale with the lowest ranking. Never even bothered listening to it.

Here is the problem you have with your argument: anyone with measurement gear can verify my findings.  No one can do that with your subjective claim.  Maybe you are right, but we don't know.  We don't know because you didn't follow any protocol to make sure you are only evaluating the sound of the device and nothing else.

We need to know for example that the knowledge of a DAC being R2R didn't influence your perception.  We need to know if  you match levels when evaluating audio gear. We need to know if you repeated your testing enough to arrive at reliable results.  We all know that in casual listening tests like yours, any and all perceptions can exist. I can listen to said R2R DAC and convince myself that it sounds like you say without said controls.  Indeed I routinely perceive differences that are not there.  We know they are not there because measurements show evidence of that, and blind controlled listening confirms the same.

This is what separates us and why ASR has Science in the middle of it.  Audio science with zero ambiguity says your evaluation of audio is faulty and without value.  When folks with those claims are tested formally, they cannot repeat their outcomes.  It is for this reason that claims like yours cannot be part of any paper for example submitted to Audio Engineering Society, ASA, etc.

As with other aspects of life, you can be a science denier and live your life happily.  Just don't put that opinion forward as an argument in mixed company.  And certainly don't use it in a thread aiming to create FUD against the other camp.