Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

Sensory experience is certainly imperfect; like all human senses. We are only human - not gods. Because we experience music from our audio systems with our senses, that sensory experience is all-important. All of it - sight, hearing, etc.

Hence the requirement for "blind" testing.

 

Blind listening for audio is a flawed practice. I don’t know anyone in pro audio that uses it. For example, at AIR studios in London, a power amplifier for their main control room (I believe) was chosen based on listening sessions. They bought a Class A/B power amp from a UK-based company called ATC.

Well except for probably most professional speaker companies. We use blind testing quite regularly in cross-over development, passive and active. There are a lot of trade-offs around crossover points, and with speakers, artifacts are audible. We used to do more blind testing around the electronics themselves, but we have a good handle on that from a measurement standpoint. We measure, we measure a lot.

 

😂Those who are fond of conducting blind tests for audio believe that a certain number of successful trials is sufficient for proving whether we can hear a difference or not... how did we arrive at this number of successful trials?!

Basic, well understood statistical functions. The more tests you do, the higher the confidence.

 

😅 8/10 or even 10/10 successful trials could be riddled with guesses and inaccurate auditory memory recall. The test subject may not admit they were unsure, because they wanted to be correct and prove their ability to be golden-eared to their peers.

Much of what you wrote is the whole point. If you are unsure and effectively guessing, that will show up in the randomness of the result. If your auditory memory is not good enough for a basic test, explain the high confidence of listening days, weeks, months apart?

 

Wearing a blindfold also creates problems that make an objective listening test more difficult. Blindfolds may hamper with the frequency response characteristics of speakers and headphones.

Um, blind testing has nothing to do with wearing a blind fold.

 

Because blindfolds are made of soft fabric with padding or a sheet of fabric, placing them over the eyes creates a sound-absorbing pocket, whereby the sound waves from speakers would not disperse as evenly with it on.

Blind testing still has nothing to do with blindfolds.

 

A blindfold may interfere with achieving a proper seal with over-the-ear headphones and on-ear headphones.

No one to my knowledge blind tests headphones.The feel of the headphone would be too obvious and you would know which is which. Defeats the purpose.

 

Lastly (for now anyway), you must acknowledge at some point in your subconcious that a "blind test" which you believe is wholly unfallible is being conducted.

Well no. For one, many here are convinced they are totally fallible. Explain why they would fail a blind test? It is really simple. You are listing to something. You don’t know what it is.

 

They have so little regard at ASR for actually listening to components that they don’t even bother listening to them. Amir performs all these tests and I don’t believe he listens to a single one or even has a reference system to drop the component in to get an impression. I read someone post here asking why the OP doesn’t take his post over there to get their reaction. Just read some of the posts there to find out. They are all a bunch of evangelical engineers who come down like a gang in mass on any one who even implies "but I like the way it sounds" just like a fundamentalist church comes down on someone questioning Adam & Eve or Noah's Ark.

I used to follow their rankings because I thought well maybe at least they would identify a component that is a disaster but no longer. Why, because a few months ago I started listening again to my MHDT Orchid Tube DAC which as you probably all know is an R2R ladder DAC and I was amazed at the quality of music it was producing over my delta sigma DAC’s, especially with acoustical music like traditional jazz which I mostly listen too. Out of curiosity I looked on the ASR site and Amir had ranked the MHDT Pagoda DAC, which is the same as the Orchid DAC but with XLR outputs as the WORST DAC THEY HAVE EVER MEASURED. They have the Pagoda DAC at the farthest right in the red scale with the lowest ranking. Never even bothered listening to it.

What’s sad is a lot of people new to audio go there to get recommendations. I really don’t care if engineers who can’t hear differences in components use the ASR guide for purchases. But I do care that some newer audiophiles who may be able to hear the differences never get a chance until it’s too late. And worse, ASR does wield a lot of power and influence and can hurt a company producing good audio products. I notice now that MHDT is no longer sold domestically in the US through a US distributor as it used to be. You now have to order one directly from MHDT in Taiwan.

That’s not how it works. Of course they measure the audio products they build - during and after the design process.The most important things are a waveform of the output stage (null test), measuring with an oscilloscope, and a select few measurements that go beyond just the standard 5 or so. SINAD is an outdated way to measure audio equipment. Yet it is used as a gold-standard on ASR.

Can you tell what those 5 select other measurements? You appear to put yourself forth as an expert, so as opposed to a vague statement, should you not detail that information?  I mainly go to ASR for speaker reviews. Professional interest. SINAD is not even part of those measurements, probably because speakers don't make noise. Because they are using a Klippel system, their measurement suite is far more extensive than say Stereophile. What is published is extensive and is similar to what we would measure internally. We have proprietary weighting functions for some measurements. I would not expect a review site to have that.  I counted 11 graphs on a DAC review, in addition to SINAD. I picked a DAC since that seems to be most discussed here.

 

Common sense tells us that for a hundred bucks, we shouldn’t be able to get a DAC with superlative performance, but ASR (Audio Science Review) tells us of course we can!

Whose common sense?  Asian manufacturing costs, Asian parts costs, a DAC a model or two down from the top, high volume, low margin business model. In our speakers with digital in, the DAC section, with enough performance to have no audible impact, is not expensive. How do we know there is no audible impact? Measurements and blind listening tests.

 

The word "Science" in the website should hint at a hypthosesis for why audio gear meant for the same purpose sounds different; and should welcome 3rd party testing - like other real scientists.

Have you noticed that ASR is publishing reviews including extensive measurements from people other than ones done by Amir. The science in the website would indicate that proving something sounds different would be a necessary first step.

 

However, that is not allowed over there...just try to challenge the results - suggest further measurements.

Here I will agree. Amir can be quite defensive and arrogant. He is not as open minded as he should be.

 

Open the device up. Take a picture of internals and indentify the parts used. Do a reliability test. None of those things are done....not to mention countless errors in testing.

Do reliability testing? Can you name even one review website that does reliability testing? Do you have any idea of the cost and time required?  I saw on several reviews pictures of internals. Not all, but a lot. What percentage of audiophiles are able to accurately assess the internals of a product?

Most of the speaker tests do not have errors. Some of the more esoteric speakers I feel have errors in testing. Some of the claims errors I have seen are more sour grapes. Given the volume of testing, errors are to be expected. Do you think other test sites are perfect? Some of the explanations and tests I see done by others, especially with lesser equipment make me shake my head.

 

Totally different impressions and MEASUREMENTS on Head-Fi for the same product. @amir_asr likes to suggest that his "instrumentation" is so much more accurate than what others are using.

Have you noticed that numerous reviewers are popping up and using guess what, the exact same equipment as ASR. Imitation is the best form of flattery I guess. His equipment does appear superior to most traditional sites. Headphone testing is very hard to do repeatedly to address that specific issue.

 

Well with that logic, upgrade every 3 months or whenever AP releases a new flagship audio analyzer. This means that every former product was substandard or less accurate in some way.

This is not a logical statement. As the test gear currently is accurate enough to identify artifacts we may hear, further accuracy only plays to marketing specifications, not audible artifacts. What is required now, is better tools for interpreting measurements, not better measurements.  For speakers, there is room for improvements for measuring distortion over emission angle, but that applies to everyone.

 

And the way he EQs headphones is painful to see. It makes me furious. He simply drags up/down a line on a log EQ so it inherently influences the frequencies around that octave as well; rather than fine-tune with proper notches in place and compensate with a preamp option in the software so the levels are not compromised. You’re welcome @amir_asr

I will leave it up to others to make their determination if your statement is true of not.

Interesting take. I’ve heard meteors as well (one over Tampa, FL around 1973 that was seen and heard by tens of thousands of people that night and I was at an outdoor concert - the meteor was LOUD).

But, yeah, “SCIENCE!” said…

NOW do a little online searching for validity of eyewitness accounts of accidents.

It’s really simple, right? Four people (count two of them - the drivers - as participant/observers too) witness a collision at a 4-way intersection.

Which story is correct? Police and claims adjusters and courtrooms are often employed to adjudicate these things because people’s memories are fungible things; what I see isn’t necessarily what you see based on your perspective and focus and abilities to recall what you saw (or think you did).

It’s been 60 years since JFK was assassinated, and eyewitnesses insisted on shots from other areas of Dealy Plaza and the grassy knoll and people are still arguing about it today. 

Basic measurements are only a benchmark, an objective standard, but how something SOUNDS is purely subjective and has to take into account intangibles like combined elements in the system, the room acoustics, speaker placement, and the listener, right?

A lot of this boils down to simple opinion, and everyone has one of those.

Amir does Amir, you do you.

@1extreme 

Right on.