2. misc-audio
  3. 175682-1635

Amir and Blind Testing

Let me start by saying I like watching Amir from ASR, so please let’s not get harsh or the thread will be deleted. Many times, Amir has noted that when we’re inserting a new component in our system, our brains go into (to paraphrase) “analytical mode” and we start hearing imaginary improvements. He has reiterated this many times, saying that when he switched to an expensive cable he heard improvements, but when he switched back to the cheap one, he also heard improvements because the brain switches from “music enjoyment mode” to “analytical mode.” Following this logic, which I agree with, wouldn’t blind testing, or any A/B testing be compromised because our brains are always in analytical mode and therefore feeding us inaccurate data? Seems to me you need to relax for a few hours at least and listen to a variety of music before your brain can accurately assess whether something is an actual improvement.  Perhaps A/B testing is a strawman argument, because the human brain is not a spectrum analyzer.  We are too affected by our biases to come up with any valid data.  Maybe. 

chayro
Post removed 
Ag insider logo xs@2x
cleeds
5,960 posts
 
noske

 

If in fact there is a difference and 40% say there is none, this is saying that those 40% of people have less than optimal hearing.

The only thing your test would reveal is that under those test conditions, 40 percent of the subjects heard no difference. There’s no data to support your conclusion.

In fact, conducting a scientifically valid double-blind listening test is not as simple as some suspect. So if the test results are puzzling, you'd also have to consider whether the problem is with the test itself rather than the test subjects.

 

 

noske
783 posts
 

The only thing your test would reveal is that under those test conditions, 40 percent of the subjects heard no difference. There’s no data to support your conclusion.

Under strict test conditions. This is a given. I do say that this isn’t easy.

Common sense suggests that something may be learnt from a test when there is a difference and 40% says there is none.  It is not a "dead" number - to an analyst it speaks information.

What would be more interesting (to some, anyway) is where there is no difference and 40% (or 60% or even just 5%) said there is a difference. Could any conclusions be drawn from this? Perhaps the test wasn’t blind or conducted properly (this includes a person who is partial to the outcome conducting the test)? Hmmm. Correctly, this aspect is conceded in the comment.

Given that ears are apparently to be trusted. As is often advocated by many good folk here.

In any event, individual tests are preferred for reasons.

nonoise's avatar
nonoise
10,887 posts
 

ASR is pretty used to empty responses like that one. It basically says "I don’t actually have any good, civil arguments or evidence in response to ASR’s reviews...but since I still don’t like their conclusions...here’s a disparaging meme so I can feel like I got one over on them."

 

Embarrassing enough once. But..3 times?

Posting something like this which is the opposite of what’s in the link, imbues you with what, pray tell? A silly sense of accomplishment?

All the best,
Nonoise

Ag insider logo xs@2x
cleeds
5,960 posts
 
noske

 

Common sense suggests that something may be learnt from a test when there is a difference and 40% says there is none.

Perhaps. It may mean that 40 percent of the test subjects simply weren’t sufficiently interested in the test to really concentrate. (If you’ve ever been a subject in one of these tests, you likely found it tedious, at least after a while. Most subjects don’t find things like ABX listening tests to be fun.)

Perhaps the 40 percent of the subjects who heard no difference were distracted by the sirens of the passing fire trucks. Perhaps the appetizers set out to keep everyone comfy and nourished triggered gastric problems. Maybe the HVAC system wasn’t keeping up with the summer’s heat as the day wore on.

Please understand that I’m not dismissing the value of scientifically valid double-blind listening tests. I’ve been part of a few and they were interesting. But analyzing the data can be tricky business.

More to discover