Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)
About 25 years ago I was inside a large room with an A-frame ceiling and large skylights, during the Perseid Meteor Shower that happens every August. This one time was like no other, for two reasons: 1) There were large, red, fragmenting streaks multiple times a minute with illuminated smoke trails, and 2) I could hear them.
Yes, each meteor produced a sizzling sound, like the sound of a frying pan.
Amazed, I Googled this phenomena and found that many people reported hearing this same sizzling sound associated with meteors streaking across the sky. In response, scientists and astrophysicists said it was all in our heads. That, it was totally impossible. Why? Because of the distance between the meteor and the observer. Physics does not allow sound to travel fast enough to hear the sound at the same time that the meteor streaks across the sky. Case closed.
ASR would have agreed with this sound reasoning based in elementary science.
Fast forward a few decades. The scientists were wrong. Turns out, the sound was caused by radiation emitted by the meteors, traveling at the speed of light, and interacting with metallic objects near the observer, even if the observer is indoors. Producing a sizzling sound. This was actually recorded audibly by researchers along with the recording of the radiation. You can look this up easily and listen to the recordings.
Takeaway - trust your senses! Science doesn’t always measure the right things, in the right ways, to fully explain what we are sensing. Therefore your sensory input comes first. You can try to figure out the science later.
I’m not trying to start an argument or make people upset. Just sharing an experience that reinforces my personal way of thinking. Others of course are free to trust the science over their senses. I know this bothers some but I really couldn’t be bothered by that. The folks at ASR are smart people too.
amir_asr
I'm all for objectifying audio if it is possible and correlates to what we hear. Speaker FR is one example of that. May I ask what are your thoughts on listening as a way to evaluate a component? Do you ever listen to some of the components you measure and wonder why you hear a difference and why measurements don't explain it? Why something may measure good and sound bad? This is not an attack on you. But internet may come across that way. I am Just curious. |
Hence the requirement for "blind" testing.
Well except for probably most professional speaker companies. We use blind testing quite regularly in cross-over development, passive and active. There are a lot of trade-offs around crossover points, and with speakers, artifacts are audible. We used to do more blind testing around the electronics themselves, but we have a good handle on that from a measurement standpoint. We measure, we measure a lot.
Basic, well understood statistical functions. The more tests you do, the higher the confidence.
Much of what you wrote is the whole point. If you are unsure and effectively guessing, that will show up in the randomness of the result. If your auditory memory is not good enough for a basic test, explain the high confidence of listening days, weeks, months apart?
Um, blind testing has nothing to do with wearing a blind fold.
Blind testing still has nothing to do with blindfolds.
No one to my knowledge blind tests headphones.The feel of the headphone would be too obvious and you would know which is which. Defeats the purpose.
Well no. For one, many here are convinced they are totally fallible. Explain why they would fail a blind test? It is really simple. You are listing to something. You don’t know what it is.
|
They have so little regard at ASR for actually listening to components that they don’t even bother listening to them. Amir performs all these tests and I don’t believe he listens to a single one or even has a reference system to drop the component in to get an impression. I read someone post here asking why the OP doesn’t take his post over there to get their reaction. Just read some of the posts there to find out. They are all a bunch of evangelical engineers who come down like a gang in mass on any one who even implies "but I like the way it sounds" just like a fundamentalist church comes down on someone questioning Adam & Eve or Noah's Ark. I used to follow their rankings because I thought well maybe at least they would identify a component that is a disaster but no longer. Why, because a few months ago I started listening again to my MHDT Orchid Tube DAC which as you probably all know is an R2R ladder DAC and I was amazed at the quality of music it was producing over my delta sigma DAC’s, especially with acoustical music like traditional jazz which I mostly listen too. Out of curiosity I looked on the ASR site and Amir had ranked the MHDT Pagoda DAC, which is the same as the Orchid DAC but with XLR outputs as the WORST DAC THEY HAVE EVER MEASURED. They have the Pagoda DAC at the farthest right in the red scale with the lowest ranking. Never even bothered listening to it. What’s sad is a lot of people new to audio go there to get recommendations. I really don’t care if engineers who can’t hear differences in components use the ASR guide for purchases. But I do care that some newer audiophiles who may be able to hear the differences never get a chance until it’s too late. And worse, ASR does wield a lot of power and influence and can hurt a company producing good audio products. I notice now that MHDT is no longer sold domestically in the US through a US distributor as it used to be. You now have to order one directly from MHDT in Taiwan. |
Can you tell what those 5 select other measurements? You appear to put yourself forth as an expert, so as opposed to a vague statement, should you not detail that information? I mainly go to ASR for speaker reviews. Professional interest. SINAD is not even part of those measurements, probably because speakers don't make noise. Because they are using a Klippel system, their measurement suite is far more extensive than say Stereophile. What is published is extensive and is similar to what we would measure internally. We have proprietary weighting functions for some measurements. I would not expect a review site to have that. I counted 11 graphs on a DAC review, in addition to SINAD. I picked a DAC since that seems to be most discussed here.
Whose common sense? Asian manufacturing costs, Asian parts costs, a DAC a model or two down from the top, high volume, low margin business model. In our speakers with digital in, the DAC section, with enough performance to have no audible impact, is not expensive. How do we know there is no audible impact? Measurements and blind listening tests.
Have you noticed that ASR is publishing reviews including extensive measurements from people other than ones done by Amir. The science in the website would indicate that proving something sounds different would be a necessary first step.
Here I will agree. Amir can be quite defensive and arrogant. He is not as open minded as he should be.
Do reliability testing? Can you name even one review website that does reliability testing? Do you have any idea of the cost and time required? I saw on several reviews pictures of internals. Not all, but a lot. What percentage of audiophiles are able to accurately assess the internals of a product? Most of the speaker tests do not have errors. Some of the more esoteric speakers I feel have errors in testing. Some of the claims errors I have seen are more sour grapes. Given the volume of testing, errors are to be expected. Do you think other test sites are perfect? Some of the explanations and tests I see done by others, especially with lesser equipment make me shake my head.
Have you noticed that numerous reviewers are popping up and using guess what, the exact same equipment as ASR. Imitation is the best form of flattery I guess. His equipment does appear superior to most traditional sites. Headphone testing is very hard to do repeatedly to address that specific issue.
This is not a logical statement. As the test gear currently is accurate enough to identify artifacts we may hear, further accuracy only plays to marketing specifications, not audible artifacts. What is required now, is better tools for interpreting measurements, not better measurements. For speakers, there is room for improvements for measuring distortion over emission angle, but that applies to everyone.
I will leave it up to others to make their determination if your statement is true of not. |
Interesting take. I’ve heard meteors as well (one over Tampa, FL around 1973 that was seen and heard by tens of thousands of people that night and I was at an outdoor concert - the meteor was LOUD). But, yeah, “SCIENCE!” said… NOW do a little online searching for validity of eyewitness accounts of accidents. It’s really simple, right? Four people (count two of them - the drivers - as participant/observers too) witness a collision at a 4-way intersection. Which story is correct? Police and claims adjusters and courtrooms are often employed to adjudicate these things because people’s memories are fungible things; what I see isn’t necessarily what you see based on your perspective and focus and abilities to recall what you saw (or think you did). It’s been 60 years since JFK was assassinated, and eyewitnesses insisted on shots from other areas of Dealy Plaza and the grassy knoll and people are still arguing about it today. Basic measurements are only a benchmark, an objective standard, but how something SOUNDS is purely subjective and has to take into account intangibles like combined elements in the system, the room acoustics, speaker placement, and the listener, right? A lot of this boils down to simple opinion, and everyone has one of those. Amir does Amir, you do you. |
