Why is Double Blind Testing Controversial?

Some years ago I agreed to test prescription eyeglasses for Pearl Vision. Several pairs were switched while I kept my eyes tightly closed. I immediately choose the "perfect" pair, that were very reasonably priced.

After a day I had a horrible headache and was forced to keep my eyes closed to enjoy my new glasses.

I like to shop for audio this way too. Make a decision based on a quick test, make myself miserable and then quit listening so the headache goes away.

Tobias and Redkiwi hit it on the head. It takes prolonged exposure (post burn-in, which a lot of A/B's may not do) to really absorb the differences. "Night and day differences" aside, most A/B's are a matter of trade-offs -- e.g., a little more bass punch vs. a little less definition. What seems very nice in the short term may not be preferred longer term. What seems like a strength listening to one recording may not seem so while listening to another. In reality, all of our choices are A/B. We are replacing what we have with something else we believe is "better" after prolonged listening. That's why companies offer 30 day trials rather than 30 minute trials. From my understanding, most A/B tests tend to be 3 minute trials. Sorry, I'm just not that good.

By the way, none of this is to say that I don't harbor a secret fear that I'd pick Bose ;-) But, I have heard too many say that, with prolonged exposure, they can pick out component signatures from a distance without seeing the actual unit/cable being used.

Well, you don't need to use the blind test to make your decision. The idea is to isolate, at least for the period of the test, the contribution of knowing who the manufacturer is, what the product costs, what it looks like, and so forth. Getting those variables out of the way at some point during your evaluation, even briefly, might be helpful, don't you think? Doesn't mean you won't choose the higher priced product in the end, but at least you will have the benefit of some calibration between what you hear and what you perhaps expect or hope to hear.

We've covered this before. As such, all i'll add to this thread is that Craig and Albert cracked me up. If DBT threads are going to be good for laughs like that, bring them on : ) Sean
>

The use of blind and double-blind procedures presumes one is employing the logic of hypothesis testing. That is, that there is a null hypothesis (i.e., that there are no differences between two treatments—in this case, two sets of interconnects) and an alternate hypothesis (there is indeed a difference). Experimenters are more than experimental custodians. Their biases and expectations can profoundly influence a study. To the extent that all people (including experimenters) have biases, one would double-blind the treatments to reduce among other things "experimenter effects." It’s surprisingly easy for an experimenter to influence a study (e.g., Stanley Milgram’s famous obedience studies). It is also easy for other participants (formerly known as “subjects”) to influence each other (e.g., Ash’s line judgment experiments where participants tended to agree with Ash’s confederates that clearly dissimilar lines were the same).

There is a famous researcher/psychologist/statistician by the name of Robert Rosenthal who once told his students that he had obtained two breeds of rats from another famous researcher. One type of rat was called “maze smart” and the other was “maze dull.” Dr. Rosenthal asked the students to teach these rats to run though mazes (ah, the power of cheese). After a few weeks or so the students were asked to show off their rats’ maze prowess (as it were). The “maze smart” rats performed significantly better than their “dumb” counterparts. The kicker here is that the rats were OF THE SAME SPECIES. One cannot infer that the students intentionally influenced the training, but it most certainly was measurable. Moreover, when the experimenter bias was measured it turned out that the “smart" rats owners had "imparted" a greater positive measurement bias than did the “dumb” rats owners negative measurement bias.

There are probably much better examples than these, but I’m in a hurry to go downstairs for dinner :-) so I’ll wrap this up soon.

Something else to consider is that “different” does not mean “better.” People’s ability to remember sounds and colors varies greatly but rarely is the memory accurate after a short decay period. With audio equipment evaluation, it tends to result in a bias for a certain “sound” regardless of whether or not that sound is authentic. When it comes to making a decision as to whether one component is better than another, it probably makes the most sense to have a reference. In the case of audio, I’d say that reference should be THE REAL THING. It’s not practical to have live orchestra tag along on equipment tests but it doesn’t hurt to keep that in mind. Some people go on and on about how they prefer one cable to another because their favorite is “warm” or whatever. Real sounds from an orchestra or a band are not necessarily “warm.”

All that said, if one believes that a $6,000 set of interconnects sounds better (they just might sound *different*) than a $70 pair then let ‘em. The more expensive cable might even sound closer to reality. One would hope that the more expensive cables aren’t just mostly cosmetics and markup.

--Paul

p.s. and yes, spending time with a set of cables or anything else in the system is a great way to know if one really likes the sound. On a marginally related note, a friend of mine once said “I’ve never owed a handheld device that I liked after having it for a week.”