Why Do So Many Audiophiles Reject Blind Testing Of Audio Components?
Because it was scientifically proven to be useless more than 60 years ago.
A speech scientist by the name of Irwin Pollack have conducted an experiment in the early 1950s. In a blind ABX listening test, he asked people to distinguish minimal pairs of consonants (like “r” and “l”, or “t” and “p”).
He found out that listeners had no problem telling these consonants apart when they were played back immediately one after the other. But as he increased the pause between the playbacks, the listener’s ability to distinguish between them diminished. Once the time separating the sounds exceeded 10-15 milliseconds (approximately 1/100th of a second), people had a really hard time telling obviously different sounds apart. Their answers became statistically no better than a random guess.
If you are interested in the science of these things, here’s a nice summary:
So reliably recognizing the difference between similar sounds in an ABX environment is impossible. 15ms playback gap, and the listener’s guess becomes no better than random. This happens because humans don't have any meaningful waveform memory. We cannot exactly recall the sound itself, and rely on various mental models for comparison. It takes time and effort to develop these models, thus making us really bad at playing "spot the sonic difference right now and here" game.
Also, please note that the experimenters were using the sounds of speech. Human ears have significantly better resolution and discrimination in the speech spectrum. If a comparison method is not working well with speech, it would not work at all with music.
So the “double blind testing” crowd is worshiping an ABX protocol that was scientifically proven more than 60 years ago to be completely unsuitable for telling similar sounds apart. And they insist all the other methods are “unscientific.”
BTW have you noticed that all of dletch2's posts have been removed ....at least from this thread. I didn't ask for it. Quite honestly I'm inclined to let people like him to continue to show themselves. If they don't self destruct they loose all credibility.
But then too, there are far to many measurebators with dogmatic opinions who have not developed the "agree to disagree mentality. No board can tolerate very many of them and we are getting more than our share.
That’s funny. Because I had no idea how cars work and so signed up for Auto Shop to learn. Every day the "class" was the teacher blathering some nonsense about maybe try and not skin your knuckles up too much today, I did that one time the wrench slipped boy did it hurt, yadayada blah blah freaking blah.
So I asked every kid there. They are all madly turning wrenches changing the carbs out and blathering banter. Not a one of them in the whole class the whole time had the foggiest idea how a car works! They were all monkeys with a hammer trying to fix a helicopter. Including the teacher!
So when I got a motorcycle and needed to tune it up I got a manual and learned myself. Two years later with a 240Z pulled the engine tore it down took it to a machine shop put it back together and it started on the first turn of the key, just fired up like it had never been taken apart. Even the timing was only very slightly off, because I had set it by eye very close which I was able to do because by this point I understand very well exactly what I am doing, how every single little bit of it works. All the stuff no one else was able to tell me about even in a class where that is supposed to be all they do.
If you want to learn this stuff the odds of running into someone like me who really does understand and can actually explain it are slim to none.
What that means is if you utter the words "double blind" in anything other than mockery and derision then YOU are the ape trying to fix the helicopter with a hammer. YOU need to drop the BS step away from the keyboard go out and DO and HEAR and LEARN- on your own.
I know it is scary having to actually do something. Yes it is a whole lot easier to jabber away on a keyboard pretending to know. But you know what? We can all clearly see that you do NOT know, because otherwise you would never utter the words double-blind in anything other than mockery and derision.
^^^artemus, there are those too. A few life examples:
guys tells me bbq cant get any better than his local guy, but has never tasted product from a true pit master
me telling my wife, the wine is costco is all I need, wife pays for a wine tasting in Paris selected by an excellent sommelier, I eat crow
the people who try and lump all things in one bucket dont know there are truly fine things in this world. Beer, cars, shoot, I even learned to appreciate a good chair recently.
I think you are right about exposure. They may not even have what we would call a system. But i suspect they’re often young and maybe have some electronics exposure and want to impress everyone with their knowledge. They remind me of the kid who had taken auto mechanics in school & just got his first car. Then he goes to the drag strip with his new 4 banger and tells the racers with 30 yrs experience how they are wrong.
@penguinpower
but its good to give some push back. the nerds wanna take over the house. not on my watch lol
That’ s partially why I started this thread. Far too many newbies here who are too dogmatic about their measurements dogma. There is plenty of evidence showing its inadequacies. Paul McGowan and many others have given good info on it. But the new know it alls don't accept it
Chill out folks. Nobody is going to come knocking to blind test your stuff without permission. If they do give them the boot. It’s your domain and your right to prohibit any practice you like. Let the test police find some other poor victim to harass! Does it ever end?
the
biggest irony with these testing nerds is that they are usually the
ones with the least exposure to a wide and varied spectrum of
equipment ...
I don't know if that's true or not, but I've long suspected it.
Its not even a question of affordability, but in most cases, its a dogmatic view of the hobby and refusal to listen.
Yes, exactly. And what we see recently on A'gon are proselytizing fundamentalist measurementalists - a noisy few - trying to reduce every discussion to blind testing. It's absurd. (And this thread has shown that science recognizes blind testing is not infallible even when conducted by experts - and the measurementalists here are obviously not experts.)
Ive been at this long enough to see all sorts. The most common zealot is the one motivated by the desire to compensate for his own insufficiencies. Whatever the circumstance, he is unable to experience variety.
That said, Ive found one right here. The guy is on a cable forum screeching about cables. The engagement is his reward. “I stuck it to them”
the biggest irony with these testing nerds is that they are usually the ones with the least exposure to a wide and varied spectrum of equipment. Its not even a question of affordability, but in most cases, its a dogmatic view of the hobby and refusal to listen. that being the case, they fail the number criteria in scientific methodology. they fail to observe. If you listen to a lot of music, and over the years have experience a good assortment of equipment, trust me, trust your ears. audio reproduction is no different from food or wine. Its is consumption that stimulates the senses. Its basis is science, but satisfaction and fulfillment comes down to giving the listener an experience.
if you want plain sustenance, Mcdonalds is right there for you. Its got everything you need and it measures well :)
djones51, and others, I wouldn't say that blind testing is a test methodology, but, rather, an experimental design control for confounding variables. Another option is to include those variables in the analysis and test for their effects directly. I sometimes do this with demographic factors such as participants' age, gender, level of education, cultural identity and such. But, all of this aligns with the multivariate research methodologies I like to use, which correlate information across multiple products and participants. Comparing only two things necessarily confounds any potentially relevant variables.
Exactly. And given that this is a hobbyist’s group and not a scientific forum, the nonsense and insults from the fundamentalist naysaying measurementalists here is really getting old and becoming an obstacle to conversation.
Which is their wont. They are, after all, zealots.
dletch2 ... Just because you seem to find solace in double blind tests doesnt mean that our observations without are any less viable. Those of us who have been around the block more than once are comfortable with our decisions and your validation mean nothing. At least not to me. Nothing helps critical listening more than experience and detailed comparisons between components sighted or not.
Exactly. And given that this is a hobbyist’s group and not a scientific forum, the nonsense and insults from the fundamentalist naysaying measurementalists here is really getting old and becoming an obstacle to conversation.
Please look back at PS Audio's video near the top of the discussion. It is right on the money. Now, some things are not easy to compare, like certain tweaks, acoustic treatments, etc., and correcting the volume for different outputs of components being tested is not easy (or may be impossible as louder typically sounds better), and sometimes other factors come into play, like how good a deal you got, the dealer making it easy with a trade in credit or handling of the sale of your old gear, etc. This is typically not black and white, and certainly not life or death.
Bottom line, if you don't like what you bought, hopefully it has some fans and you can sell it here or on another site.
Lets stay away from exaggerated comments like cults that have negative (or are negative) connotations. Enjoy the hobby and the gear whether you keep things for a long time or a constant tinkerer with upgradeitis.
Your comments are very enjoyable but I was wondering if most of these comparative tests are done blind, at least as much as possible. If not, how do you control for bias? Are there other techniques besides blind testing you could use for difference in sound?
So even when dletch2 does his informal blind tests, he only proves that in his system and at that moment his taste says that one component betters another.
Indeed, what has been proven? It seems that some here want to get away from all biases as if that is even possible? Suppose the guy next to you or the operator gives an approving gesture or says "Wow!" every time he likes the sound? Then what bias kicks in? We are human, different & unique, Not machines. But even machines often have bias
Seriously dltech2, you have to come up with something better than cults. You are wasting my time with these stretches. You are the one asking us to join the blind listening cult and telling us that without this method we are being duped. We are not asking you to change anything in your regimen we just propose that it doesnt accomplish what you think in all cases and are not asking you to join our cult. Make no mistake, as per your definition, these are both cults apparently.
I Think he believes that its just a matter of putting on a blindfold. From what I have read, there is a lot more to it and really is not worth the time considering the results.
What they think they hear is the same as what they hear!
dletch2, You are in no position to make this judgement. This is my whole point. Just because you seem to find solace in double blind tests doesnt mean that our observations without are any less viable. Those of us who have been around the block more than once are comfortable with our decisions and your validation mean nothing. At least not to me. Nothing helps critical listening more than experience and detailed comparisons between components sighted or not.
So lets whip em out, place them on the table and compare. Give me your background in terms of 2 channel and your current system specifics. Analog or digital, etc. Also list room treatments, dimensions and any other relevant information like age. If you want to include degrees and your profession that is fine.
Please stop trying to place all enthusiasts in the same small box simply for your peace of mind. And stop ascribing the same motivations and biases to all as if this were some cosmic constant.
Yes it’s a very imperfect world out there but Audiogon members on threads are working hard to somehow make it better. Thank G-d they got their hifis and what they think they hear to help stay sane.
Well after you buy something what you choose to do is up the the individual and I feel it no more likely with an expensive product than a cheaper one. We see clues of this everywhere on this forum with suggestions that a Magnaplanar or horn speaker is better than the Wilson or other types. I see nothing wrong advertising goals because questioning proper etiquette removes individual responsiblity from the experience. You might say that the person who succumbs to advertising and buys a very expensive speaker was manipulated but I would never presume to know the persons motivation. Nor that they were inspired some dletch2 bias or for any other reason that it sounded better. Why dont some of you admit that there is a huge moral piece to all of this in terms of levels of accepted consumption (see Wilson thread).
Seriously dltech2, you have to come up with something better than cults. You are wasting my time with these stretches. You are the one asking us to join the blind listening cult and telling us that without this method we are being duped. We are not asking you to change anything in your regimen we just propose that it doesnt accomplish what you think in all cases and are not asking you to join our cult. Make no mistake, as per your definition, these are both cults apparently.
Me again. I commented above on the psychometric technique of Semantic Differential rating scales, which is the primary technique I use in assessing perception because it is easy for participants to use and it differentiates products along multiple dimensions simultaneously. However, I have also used another psychometric technique, which is based upon similarity judgements. It strikes me that this touches upon the issue of AB and ABX testing. I collect similarity data either by having participants in my studies rate the degree of perceived similarity between pairs of items, or with a "triad" method in which items are presented three at a time and the participant selects the one that seems most different. Each triad provides input to three cells of a (dis)similarity matrix. Cells for the two items that weren't chosen as being "different" are coded "0" for minimum dissimilarity and cells for the two pairings involving the "different" item are coded "1" for maximum dissimilarity. I sum the matrices for individual participants and analyze it using Multidimensional Scaling to map all items within a multidimensional perceptual space.
The rating method provides what is essentially a continuous metric for the AB comparison and the triad method provides something that is functionally similar to ABX except with three different items and the question being reversed, i.e., not which two are the same, but which one is most different? The two items remaining after the odd-man-out become the "same" items in ABX.
Although many psychometricians assume that the similarity judgments and Semantic Differential ratings produce equivalent results, I find that is usually not the case. In my experience, the Semantic Differentials (using Factor Analysis) differentiate a larger number of independent perceptual dimensions whereas the similarity comparisons (using Multidimensional Scaling) sometimes reveal higher order perceptual qualities that are not easily described with words. Either way, the emphasis is on identifying the number of different ways that things vary. Perception is always multidimensional unless you intentionally restrict the range of variation. Ironically, that is exactly what happens when you compare only two or a small number of items.
I believe he was talking about the point of advertising--advertisers use our own nature (and pride) to sell their products, making us believe they are better, more luxurious, more this more that. Some products may be better, and I think my post above specifically pointed out that comparison of different models and coming away liking whatever it is you picked out is a good thing. What’s not good is when you buy it and then lord it over others (or when you don't and then berate others who did). Not sure why there is an issue with that.
Well how can you fault a business for trying to manufacture a market? And it is different in that you can buy a product and decide for yourself, using whatever methods you choose, if the item is worth the asking. A tangiable product which you can audition and compare against other similarly priced products. It isnt even remotely the same thing and you suggesting it is similar suggests that you are simply another casualty of this age.
Setting up a proper blind test is very difficult and must address several issues while other issues with the test cant be controlled. You must guarantee the precise volume and you must have the ability to switch very quickly (seconds).
One of my friends was flown to participate in one of the Harmon tests involving, I believe, a new Infinity speaker. These new speakers were up against B & W 801s. Output was identical with 3 to 5 second snippets of the same music played between the 2 speakers with preferences tallied by the listeners 20 times. I think that he said it was all in mono and he was the only person to pick the B & W every time.
When I asked him how it was he said that it was actually somewhat stressful and he thought the test flawed on numerous levels. First and foremost he thought that the upstream gear had a significant impact on the sound of the speakers and that this aspect only proved which system configuration sounded better at that moment.
So even when dletch2 does his informal blind tests, he only proves that in his system and at that moment his taste says that one component betters another.
Interesting thread. Just my two cents, but don’t we all listen in order to choose? I prefer that if I am listening to a couple different things and trying to decide between them, I would rather that I not know which I am listening to in order to take away any bias--I think we all know it works, even if we don’t have graphs etc to back up our findings.
In the end, isn’t is just about choosing what we like to listen to? Even if a person did a AB comparison or a ABX or name your comparison, isn’t that just what sounds good to them? Doesn’t mean I have to like it--maybe I like those $500 speakers instead of the $5000 ones; maybe they are the opposite. I have a friend who likes Marmite, and I think it tastes disgusting. Regardless of any testing, I am never eating that stuff again, even if he tells me over and over that I should like it.
The never ending quest to show that what you have or like is better than what another person likes or owns, or to tell another person that they are morons for purchasing x or y is nothing but pride. It just gets in the way of a community of listeners enjoying the hobby and turns the whole thing into an adversarial mess.
Testing is cool and all. But seriously, this is something that manufacturers should do as part of R&D.
Im pretty sick of the garage audio scientists putzing around with their 75 dollar mics and charts claiming authority over the audio universe. I have a life. I spend my spare listening to music and fiddling with my various hobbies. IDGAF about condescending zealots.
@nonoise, I am not sure what "aha" you think @jakleiss has introduced to this topic? Nothing in what he wrote suggest that blind testing is a bad idea.
Is that some more of your remote viewing capabilities?
You must have a verified phone number and physical address in order to post in the Audiogon Forums. Please return to Audiogon.com and complete this step. If you have any questions please contact Support.
