Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

I will even go so far as to say that I do agree with a lot of what @amir_asr says.  Our subjective sensory experience IS flawed.  We’d be lying to ourselves if we thought that wasn’t the case.  I’ve been caught with that, with initial impressions of a new piece sounding way better, only very quickly realizing later that it wasn’t actually THAT much better.  I also agree that blind testing is valuable, whenever possible.  Even blind testing is somewhat limited.  Just because one might not be able to discern differences in a blind test does not mean that those indiscernible differences may not add up to greater or lesser enjoyment over the long term, even if we are not directly aware of those differences.

BUT.  I do believe your sensory experience, while flawed, is the best form of measurement we have and the one that ultimately matters.  The problems Amir mentions which I agree with, can be worked through, with extended listening and, as much as possible, keeping an open mind. 

As an aside.  Since @amir_asr has supported the practice and value of blind testing, why does he not do that with the gear he measures?  If it was done in a fair, controlled manner, it would be really great to see how all of those gears he trashes would fare in such a test.   It would be great to see how frequently his measurement results would be corroborated by such tests.

If I were Amir and I truly believed in my heart the value of measurements, I’d want to prove it with such tests, which Amir himself has said are useful.  It would be great to see a blind test of burnt in vs brand new gear.

That would be the ULTIMATE forum/YouTube channel.  One that specializes exclusively in controlled, double blind testing.

@nyev

Sensory experience is all we’ve got.

Go to a costly steakhouse - order a steak. Does it taste just good or much better than a $30 steak? we’re talking 3 - 6 times the price. If it is dry, not properly cooked (medium or medium rare etc. to your liking, you will notice). Even if you were blindfolded and asked to try, you would notice.

Ask a friend to audition 2 different bluetooth headphones with the same song on their smartphone. Then once they’re done, ask them which headphone they liked more and why. You could aruge they were not perfectly volume matched - but evenso, the differences in sound quality will be obvious.

Sensory experience is certainly imperfect; like all human senses. We are only human - not gods. Because we experience music from our audio systems with our senses, that sensory experience is all-important. All of it - sight, hearing, etc.

Blind listening for audio is a flawed practice. I don’t know anyone in pro audio that uses it. For example, at AIR studios in London, a power amplifier for their main control room (I believe) was chosen based on listening sessions. They bought a Class A/B power amp from a UK-based company called ATC.

😂Those who are fond of conducting blind tests for audio believe that a certain number of successful trials is sufficient for proving whether we can hear a difference or not... how did we arrive at this number of successful trials?!

😅 8/10 or even 10/10 successful trials could be riddled with guesses and inaccurate auditory memory recall. The test subject may not admit they were unsure, because they wanted to be correct and prove their ability to be golden-eared to their peers.

Wearing a blindfold also creates problems that make an objective listening test more difficult. Blindfolds may hamper with the frequency response characteristics of speakers and headphones.

Because blindfolds are made of soft fabric with padding or a sheet of fabric, placing them over the eyes creates a sound-absorbing pocket, whereby the sound waves from speakers would not disperse as evenly with it on. And since our eardrums are levelled to our eyes, they can be a direct source of sound wave absorption/reflection from the blindfold strap or covering.

A blindfold may interfere with achieving a proper seal with over-the-ear headphones and on-ear headphones. If conducting such a test, the test subject may already be attuned to how said headphone fits and feels on or over their ears... thereby giving them a sense of pseudo-preferential vision, much like figuratively having another set of eyes to visualize the set of headphones being auditioned in their mind during the test.

Lastly (for now anyway), you must acknowledge at some point in your subconcious that a "blind test" which you believe is wholly unfallible is being conducted. And as the subject of that test, you start to believe in the testing methodology. If you haven’t been the subject of such a test before, that is blind faith being invested...no better than losing teeth as a kid and asking the tooth fairy for 20 bucks.

If the researchers recorded the meteor sound and played it to audiophiles, would they all agree they heard the same thing?

@mastering92 , all valid points.  It’s a matter of figuring out, among all the flawed approaches to evaluating gear, which is the least flawed.  I agree the answer to that is unequivocally the use of our own senses.

To your point with blindfolds introducing a “probe effect”, I’m not sure if it’s in my head or not, but I feel like there could be a difference in sound when I put my glasses on.  What I know for certain is that transparency is increased when I look to the floor, as opposed to when looking straight forward.

As a scientist, one of the really cool things I get to experience a times is an observation that does not make sense or is counter to consensus.

It is important to remember that "consensus" among scientists is science by democracy - it is not scientific at all, just our current opinion and can change in an instant when new information is gained.

When I see someone point at a measurement and state boldly "that is not audible," I am reminded of the same being said about MP3 when Apple first started pushing the format. Same thing was said about jitter for many years, engineers stating boldly that human hearing is not that sensitive to time.