Amir and Blind Testing


Let me start by saying I like watching Amir from ASR, so please let’s not get harsh or the thread will be deleted. Many times, Amir has noted that when we’re inserting a new component in our system, our brains go into (to paraphrase) “analytical mode” and we start hearing imaginary improvements. He has reiterated this many times, saying that when he switched to an expensive cable he heard improvements, but when he switched back to the cheap one, he also heard improvements because the brain switches from “music enjoyment mode” to “analytical mode.” Following this logic, which I agree with, wouldn’t blind testing, or any A/B testing be compromised because our brains are always in analytical mode and therefore feeding us inaccurate data? Seems to me you need to relax for a few hours at least and listen to a variety of music before your brain can accurately assess whether something is an actual improvement.  Perhaps A/B testing is a strawman argument, because the human brain is not a spectrum analyzer.  We are too affected by our biases to come up with any valid data.  Maybe. 

chayro

It appears that the fake science site are such complete amateurs that they measured the wrong AC output port on their recent power conditioner “review” according to Paul McGowen, who said they should issue a retraction for their shoddy work.

I call it the “idiot with an analyzer” phenomenon, but psychologists have a term for this kind of limited intelligence masquerading as knowledge.

“Dunning-Kruger effect, in psychology, a cognitive bias whereby people with limited knowledge or competence in a given intellectual or social domain greatly overestimate their own knowledge or competence in that domain relative to objective criteria or to the performance of their peers or of people in general.”

 

Currently it is an LLC (Audio Science Review LLC, registered in Washington State). It is not setup as a non-profit with respect to IRS so unfortunately your donations are not tax deductible. Worse yet, I have to pay taxes on them.

Edit: I am highlighting this as there still seems to be some confusion about this.

I stand corrected. There are no ads on their forum or YouTube channel. However please see above. In other words the website does generate revenue (profit). 

First, the OP reports Amir as claiming confirmation bias in "analytical mode": "when he switched to an expensive cable he heard improvements, but when he switched back to the cheap one, he also heard improvements because the brain switches from 'music enjoyment mode' to 'analytical mode'.” If this were true, however, one could in theory continue switching out "better" for "lesser" components, experiencing an "analytical mode" confirmation bias each time, until one was left with a Walmart rig (that is, a system we would all agree sounds terrible; I'm not impugning Walmart, merely coining a shorthand). If every change I make to my system results in a subjective impression of improvement, I could never move forward at all. This is tantamount to just saying: there is no audible difference between systems. But that is obviously false. So this methodological claim is logically flawed.

Second, as one other contributor to this thread has written, the brain's memory for subtle auditory cues is extremely short. Therefore, the only way to judge whether or not a given system change causes an audible difference is by instantaneous A/B, or better still, A/B/X blind testing. Again, this well established neurological fact is inconsistent with Amir's reported claim.

Third, as has also been mentioned here already, loudness needs to be very carefully matched between A and B, which is often extremely tricky to do. Louder will almost always sound "better," even when the difference in db is so slight that it isn't perceived as a difference in volume level. This is why people claim balanced cables sound better. XLR cables are advantageous in certain circumstances, but not in any that are relevant to home audio systems; in fact, for home audio, balanced is slightly (measurably but not audibly) worse. However, balanced cables are a few db louder, and so often are perceived as sounding "better."

Fourth, we're talking such subtle differences here anyway that a host of what might be called psychoacoustic phenomena become much more relevant. It's a little like wine tasting. The chemistry of one's mouth, which varies with the food you've eaten recently, with the state of your digestion, and a lot of other things, as well as one's mood, the company one is with, and so on, will affect one's impression of a given wine, which makes it very difficult to say that a given bottle is better or worse than the very same wine tasted on a different day.

Finally, the wine analogy perhaps shows why, despite the fact that the brain has too short a memory of subtle sonic differences to make comparisons over extended listening times (hours or days or weeks), it is still that long-term experience one must try to evaluate. Although the same bottle of Napa Cab from 2016 will taste differently on different days, or at different times of the same day, it is still possible to know with confidence that I prefer the bottle of Screaming Eagle (or whatever) to the Walmart Special—or even to a wine comparable in price and reputation to the Screaming Eagle. Similarly, I know with some confidence that I prefer system X over system Y as a result of hearing both repeatedly over a long period of time. It can't be that my brain is comparing subtle differences only evident in instantaneous A/B testing, and yet it is just as surely the case that I can, and do, have psychoacoustic preferences that are consistent over long periods of time.

Measurements are great. HOWEVER... We measure what we think to measure. There are so many things going on that we don't have a measurement for. Yet some will put way too much weight on measurements. We need both. But I weight my senses as the final arbiter. I do agree our ears and brains are much more sensitive than many can imagine. For me blind tests are problematic...because our minds want to discriminate the 'outstanding' stuff...the most noticeable stuff while we may filter out nuance. Long term testing is needed to rule out mistakes and quantify the nuances. If something sounds better but you keep yearning for the other option...well, something else is going on. Possibly something we haven't a measurement for. Learn to develop and trust your senses...

End of the day let’s all agree to disagree. Some people enjoy Amir’s opinions. Others don’t.
 

I personally can’t be bothered to validate or comment on his experiments because, unlike Amir, who KNOWS he can’t be wrong because his numbers don’t lie, I have had first hand experience with some of the “exposed” products and I couldn’t disagree more with him. I don’t care what Amir thinks he KNOWS about a power plant because I have heard both P12 and P20 extensively and the difference they make in some systems is not subtle. I have heard what an audiophile switch can do even connected to cheap and cheerful streamers so, again, his lambasting falls on deaf ears.
 

My advice to his followers is simply to go out, listen, make up their own minds. Most of these products can be demoed for free, PS Audio has a generous return policy, no questions asked. Why not try a P12 for free and see whether Amir is indeed the Neo in our audiophile Matrix or the woman in the red dress? 
 

Enjoy the music🕺