Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)


About 25 years ago I was inside a large room with an A-frame ceiling and large skylights, during the Perseid Meteor Shower that happens every August. This one time was like no other, for two reasons: 1) There were large, red, fragmenting streaks multiple times a minute with illuminated smoke trails, and 2) I could hear them.

Yes, each meteor produced a sizzling sound, like the sound of a frying pan.

Amazed, I Googled this phenomena and found that many people reported hearing this same sizzling sound associated with meteors streaking across the sky. In response, scientists and astrophysicists said it was all in our heads. That, it was totally impossible. Why? Because of the distance between the meteor and the observer. Physics does not allow sound to travel fast enough to hear the sound at the same time that the meteor streaks across the sky. Case closed.

ASR would have agreed with this sound reasoning based in elementary science.

Fast forward a few decades. The scientists were wrong. Turns out, the sound was caused by radiation emitted by the meteors, traveling at the speed of light, and interacting with metallic objects near the observer, even if the observer is indoors. Producing a sizzling sound. This was actually recorded audibly by researchers along with the recording of the radiation. You can look this up easily and listen to the recordings.

Takeaway - trust your senses! Science doesn’t always measure the right things, in the right ways, to fully explain what we are sensing. Therefore your sensory input comes first. You can try to figure out the science later.

I’m not trying to start an argument or make people upset. Just sharing an experience that reinforces my personal way of thinking. Others of course are free to trust the science over their senses. I know this bothers some but I really couldn’t be bothered by that. The folks at ASR are smart people too.

nyev

When someone cannot effectively describe how something sounds in different ways, measurements alone by self-proclaimed experts are of no interest to me. 

I will even go so far as to say that I do agree with a lot of what @amir_asr says.  Our subjective sensory experience IS flawed.  We’d be lying to ourselves if we thought that wasn’t the case.  I’ve been caught with that, with initial impressions of a new piece sounding way better, only very quickly realizing later that it wasn’t actually THAT much better.  I also agree that blind testing is valuable, whenever possible.  Even blind testing is somewhat limited.  Just because one might not be able to discern differences in a blind test does not mean that those indiscernible differences may not add up to greater or lesser enjoyment over the long term, even if we are not directly aware of those differences.

BUT.  I do believe your sensory experience, while flawed, is the best form of measurement we have and the one that ultimately matters.  The problems Amir mentions which I agree with, can be worked through, with extended listening and, as much as possible, keeping an open mind. 

As an aside.  Since @amir_asr has supported the practice and value of blind testing, why does he not do that with the gear he measures?  If it was done in a fair, controlled manner, it would be really great to see how all of those gears he trashes would fare in such a test.   It would be great to see how frequently his measurement results would be corroborated by such tests.

If I were Amir and I truly believed in my heart the value of measurements, I’d want to prove it with such tests, which Amir himself has said are useful.  It would be great to see a blind test of burnt in vs brand new gear.

That would be the ULTIMATE forum/YouTube channel.  One that specializes exclusively in controlled, double blind testing.

@nyev

Sensory experience is all we’ve got.

Go to a costly steakhouse - order a steak. Does it taste just good or much better than a $30 steak? we’re talking 3 - 6 times the price. If it is dry, not properly cooked (medium or medium rare etc. to your liking, you will notice). Even if you were blindfolded and asked to try, you would notice.

Ask a friend to audition 2 different bluetooth headphones with the same song on their smartphone. Then once they’re done, ask them which headphone they liked more and why. You could aruge they were not perfectly volume matched - but evenso, the differences in sound quality will be obvious.

Sensory experience is certainly imperfect; like all human senses. We are only human - not gods. Because we experience music from our audio systems with our senses, that sensory experience is all-important. All of it - sight, hearing, etc.

Blind listening for audio is a flawed practice. I don’t know anyone in pro audio that uses it. For example, at AIR studios in London, a power amplifier for their main control room (I believe) was chosen based on listening sessions. They bought a Class A/B power amp from a UK-based company called ATC.

😂Those who are fond of conducting blind tests for audio believe that a certain number of successful trials is sufficient for proving whether we can hear a difference or not... how did we arrive at this number of successful trials?!

😅 8/10 or even 10/10 successful trials could be riddled with guesses and inaccurate auditory memory recall. The test subject may not admit they were unsure, because they wanted to be correct and prove their ability to be golden-eared to their peers.

Wearing a blindfold also creates problems that make an objective listening test more difficult. Blindfolds may hamper with the frequency response characteristics of speakers and headphones.

Because blindfolds are made of soft fabric with padding or a sheet of fabric, placing them over the eyes creates a sound-absorbing pocket, whereby the sound waves from speakers would not disperse as evenly with it on. And since our eardrums are levelled to our eyes, they can be a direct source of sound wave absorption/reflection from the blindfold strap or covering.


A blindfold may interfere with achieving a proper seal with over-the-ear headphones and on-ear headphones. If conducting such a test, the test subject may already be attuned to how said headphone fits and feels on or over their ears... thereby giving them a sense of pseudo-preferential vision, much like figuratively having another set of eyes to visualize the set of headphones being auditioned in their mind during the test.

Lastly (for now anyway), you must acknowledge at some point in your subconcious that a "blind test" which you believe is wholly unfallible is being conducted. And as the subject of that test, you start to believe in the testing methodology. If you haven’t been the subject of such a test before, that is blind faith being invested...no better than losing teeth as a kid and asking the tooth fairy for 20 bucks.

If the researchers recorded the meteor sound and played it to audiophiles, would they all agree they heard the same thing?