Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

@1extreme 

They have so little regard at ASR for actually listening to components that they don’t even bother listening to them. Amir performs all these tests and I don’t believe he listens to a single one or even has a reference system to drop the component in to get an impression.

You need to put aside "belief" and substitute reality.  I listen to huge amount of audio gear as part of my testing.  Every speaker, headphone, headphone amp and even some audio tweaks such as power conditioners, cables, etc. have listening tests.  This adds up to about 200 devices a year that get reviewed with listening tests.

I have a reference system as well where equipment is tested.  For example, I used RME ADI-2 Pro ($2000) driving a $4000 Dan Clark headphone to test a headphone amp.  Cables are tested the same way if they are interconnects and such.  Ditto for power conditioners, etc.  Here is an example from this week alone, a headphone amp: 

"This is a review, listening tests and detailed measurements of the Eleven XIAudio Broadway balanced battery operated desktop headphone amplifier. "

Far field devices like speakers get tested in and against my nearly $100K main audio system.  Power is provided by $20K in amplification for example.  I list these prices not that they should matter, but I suspect matter a lot to you as to what makes a "reference system."

But no, I don't listen to everything.  Measurements so powerfully describe the performance of such devices, showing impairments well below hearing for example, where it makes no sense for me to listen to them.  And proceed to make up stuff like subjective reviewers do.  Mind you, if measurements show a problem, I do listen even in that category.  Here is an example of that, the PS Audio DirectStream DAC: 

https://www.audiosciencereview.com/forum/index.php?threads/review-and-measurements-of-ps-audio-perfectwave-directstream-dac.9100/

"I started the testing with my audiophile, audio-show, test tracks. You know, the very well recorded track with lucious detail and "black backgrounds." I immediately noticed lack of detail in PerfectWave DS DAC. It was as if someone just put a barrier between you and the source. Mind you, it was subtle but it was there. I repeated this a few times and while it was not always there with all music, I could spot it on some tracks.

Next I played some of my bass heaving tracks i use for headphone testing. Here, it was easy to notice that bass impact was softented. But also, highs were exaggerated due to higher distortion. Despite loss of high frequency hearing, I found that accentuation unpleasant. WIth tracks that had lisping issues with female vocals for example, the DS DAC made that a lot worse."

You could have figured all of this out by simply going and looking.  But instead you just repeated incorrect talking points.

@cleeds 

Exactly. The measurements and data have little value if not correlated with what we hear.

They absolutely are correlated with what you hear based on science of psychoacoustics which is entirely based on listening tests.  Problem you all have is that you don't to hear, pun intended, the consequences of that.  Instead you want to live in a fantasy world where differences that don't exist or are below audible, are so obvious that your wife can hear from the kitchen.  That kind of made up effect is psychological and can never be caught or explained in a proper, science based, review of audio product.

@cleeds 

I really don't think ASR has much power or influence at all. It's just a noisy group with grievances, which is very common today.

Well, I know more than one company that has completely retooled their entire product development and product strategy precisely because of the work we have been doing (think Schiit).  So we do have influence, having nearly 3 million visitors a month which puts us at or at the top of all audio based sites on the Internet.

The main grievances, as evidenced by threads like this, are people who don't want to come along.  They rather cling to "beliefs" than accept reality of objective and science based evaluation of audio.  Why?  Because in some cases it negates their praise for some product.  Instead of learning from that, they choose to go after ASR.  And for what?  Us providing more data than you had before about your purchases?  There is no better definition of asking, heck demanding, to put one's head in the sand.

By all means, do that but don't make up stuff about ASR.  At least do some homework before creating FUD like this.

amir_asr

.... you want to live in a fantasy world where differences that don't exist or are below audible, are so obvious that your wife can hear from the kitchen ...

You live in a fantasy world. You don't know anything at all about me. But you have an active imagination, I'll give you that.

@normb 

Basic measurements are only a benchmark, an objective standard, but how something SOUNDS is purely subjective and has to take into account intangibles like combined elements in the system, the room acoustics, speaker placement, and the listener, right?

Yes and no.  In most cases, audio gear is designed to have its own performance or you would never be able to assemble any system.  Or trust any reviewer whatsoever, right?  For example, a source device such as a DAC has a low impedance of say 100 to 200 Ohm.  The pre-amp then has an impedance of at least 10X that.  This way you get full voltage transfer which is what we want.

On the other hand, there are tube amps with high output impedance which then interact with the frequency response of the speaker.  This causes tonality to shift in the system not because of anything good, but because of poor design.  A solid state amp will have well below 1 ohm impedance as to eliminate this effect.  You could hear these effects using either listening tests (if the person is properly trained and difference large enough), or measurements.

In vast majority of cases though, the modular aspect of audio allows us to independently test and evaluate a component by itself.  Measurements are much more powerful in this regard because audiophiles as a group are terrible at detecting non-linear artifacts. But even for things like speakers where distortions are apparent, we are a) mostly alike when it comes to preferences and b) non-professionally trained listeners including reviewers and dealers are terrible at providing consistent and proper feedback.  Please see this formal study: 

Indeed, the subjective data from audio reviewers is so bad that you need 10 times as many of them to equal properly and formally trained speaker listeners!  That is the problem with subjective remarks from audiophiles or audiophile press.  It is so unreliable that it is not worth paying attention to.  The same study by the way shows that listener preference is similar among a dozen different listener classes:

See how the ranking of each speaker did not matter (different colors) regardless of who was listening to it (X axis).  Green speaker for example was bad no matter who was evaluating it, in controlled tests that is.

If you want to see a more detailed explanation of that, I have done a video on it: