do the results of a blind or double tests have any usefulness ?
a blind ab test does not prove anything. it only states facts, namely that a certain percentage of listeners were able to distinguish the sound of one component from another.
there are many issues as to the validity of these tests.
the question, "can a listener detect the sonic difference(s) between components", is never answered definitely by one test.
suppose a blind test is administered 10 or more times to the same group of listeners. assuming the first test is replicated 9 more times, there is a chance that there may be an inconsistency between the outcomes of the test.
suppose a statistically significant percentage, say less than 60 percent, for a large sample of listeners, identifies differences between 2 components, what does that mean ?
what is the application of such a result for a prospective buyer of a component ?
How bout this. Take a blind test and place a dollar value on the entire system. That is, if the components weren't purposely mismatched to limit performance. For someone to estimate a $25,000 value on a $3000 system (or visa versa)would be quite a reality check.
Too many people base their opinions on $$$ invested. I find that irritating and simple minded.
I lied about MrTennis, he's not the one who has been posting on stereophile. Sorry Mr T.
Double blind testing is a great way to determine a listeners ability to hear, but that is about all. It's not a good way to determine the value of a product.
The small group of basement-dwelling "audiophiles" is so widely dispersed that creating a test sample large enough to produce statistically significant results is near impossible. "What about CES?" you might say. Sure you could gather a sample group and get your blind test. But the group would devolve into a small-scale riot over the type of wire used to run the lighting in the room or the composition of the ceiling tiles.
MrT, the reasons you cited are exactly why dbts are such a painful subject. Actually I should say DBTs with respect to human sensory perception, since taste tests are also notorious in many of the same ways that aurally oriented tests are. Nonetheless, I still think there is value in pursuing comparison criteria, since, in a perfect world, the DBT could be used an excellent determination of the value of a given component in the context of its peers.
Would they prove that the emporer in fact has no clothes? I don't know, but let's hope there's at least a loincloth...
To do this right is very difficult. Read the book Blink to get an idea of how humans form judgements. From psychology we know that people will preceive coffee from a red can as richer and more full bodied than the same coffee from a green can. Every single thing that is said to the subject, every thing in the enviroment, the order things are presented in, and even the lead experiementers beliefs, even when double blind designs are used influences the results.
On a related topic, it's well known that if by accident or otherwise you lose the use of one of your senses that the remaining senses become more acute. That being the case, has any audiophile out there tried poking out their eyes to improve their hearing? It's probably a bigger upgrade than any sub-$3k power cable.
Onhwy61: I agree, I think you might possibly get more bang out of say a 4k power cord than out of blinding yourself, but below 3k? No way. But I really think you have got to spend 6k on a power cord before you can say for sure you've done more for your system than you can do with self mutilation alone.
Blind tests are absolutely useful. I wish all reviewers would NOT know ahead of time the brand, cost, etc. of the component they're reviewing. I think marketing, reputation, price sets expectations that influence a review.
Double blind tests are even more useful. I would love to know if expensive components really sound better than inexpensive ones. Relative comparisons can be very useful. 80% of threads on this board start like this, "What's better X or Y?" Again a blind comparison would provide VERY objective opinions.
How about an alternative to the blind test; the nearly blind or poorly sighted test where all participants are required to have poor eye sight and must turn their glasses or contacts in before listening. This would be a politically correct compromise between the totally blind objective test and the totally sighted subjective test.
i have been thinking about this question for a while.
i had forgotten that i initiated this thread, months ago.
i think the important question is: does one prefer component "a" over component "b".
an interesting answer to this question would be to perform 2 tests. the first test would consist of comparing the components without being aware of the brand names. the second test would be identical, except that the names of the components would be revealed prior to the comparison. presumably, several months would elapse between the 2 tests.
the conditions of the test, such as sources, duration of exposure, manner of substituting one component for another, etc., could be determined by a committee of "wise men".
the question is, is there a placebo or "mind over matter" or prejudgment effect of being told what components are being compared , such that the results of the tests would be different ?
In medicine and in science in general the ONLY kind of test considered reliable is a double-blind test. Should listening tests of audio equipment be different? Perhaps it should because the "data" is completely subjective. But this means that the test results are completely useless, so why bother? Go listen.
In medicine and in science in general the ONLY kind of test considered reliable is a double-blind test. Should listening tests of audio equipment be different? Perhaps it should because the "data" is completely subjective. But this means that the test results are completely useless, so why bother? Go listen.
In the sense that a double-blind test can be used to show that a difference is inaudible even to experts....YES this is very useful to separate hyperbole from the reality about the magnitudes of difference between components.
Given two pieces of equipment that do have an audible double blind statisticaly meaningful difference, then I am with Eldartford.....the "better" sounding piece of gear is a subjective matter...some want "accuracy" which involves science and test measurements...some want the nicest sound colored to their personal preferences!
Science has no place in our hobby. We know what we know and we don't need no book learnin'.
Science and test measurements do have a place in audio, but are only relevant to a small portion of audiophiles, a much higher preportion of manufatcurers (to help design, build and QC) and to a very high proportion of sound engineers/studios.
Pawlowski6132...Reviews tell us the features of the product, like balanced/unbalanced inputs, and some basic parameters, like power rating. This is mostly why I read reviews. On a very few occasions glowing descriptions of audio quality from several sources have caused me to seek out and audition a product. But I don't buy on the basis of reviews.
From a technical point of view (I am an engineer) I find JA measurements in Stereophile interesting. I often read these, but not the associated subjective evaluation. Besides, the stuff they choose to review is usually out of my price range anyway.
Eldarford; Although I agree with your approach, I think most people gather the specifications of products from the manufacturers websites and read reviews to find out, simply put, if the gear sounds any good since dealers aren't always available.
I just happened upon this quote this morning in the February 2007 issue of "The Absolute Sound". It's a review of the Mark Levinson No. 436 Power Amp by Sue Kraft:
"So it was with much anticipation that I awaited the arrival of the Mark Levinson reviews. I was also quite excited at the prospect of hearing how these $12,500, 350Wpc brutes would mate with the stellar imaging and accuracy of a loudspeaker like the B&W 800D. Would my decade long wait to finally hear a pair of Levinson amps in my system meet my expectations? At the risk of ruining the ending...I'm afraid I knew the answer to that question nearly straight out of the box, and have heard nothing in the months to follow that would change my mind."
Come on. How can anyone argue that blindfolding Sue and telling her to review this amp would not result in a more objective review????? Wouldn't that be more helpful to the above mentioned "typical" reader I mentioned????
i think the purpose of the thread is to suggest the advantage of evaluating the sound of a component in absence of any attitudes about the component before the audition.
one does not literally have to be in a dark room or wear a blindfold. i think it is sufficient to be sighted so long as the brand and model of the component is not known.
unbiased assessment reduces the likelihood of erroneus judgments.
a blindfold is irrelevant. the purpose of a blind test is to ensure that a listener is unaware of the component auditioned. if one sees the component and it has a generic look and does not exhibit any visual cues to identify the product, the listener will be totally ignorant of the manufacturer as well as any other information about the product. blind should be interpreted in the figurative sense, rather than literally.
Not having been to Sue Kraft's new house I can't state it with absolute certainty, but I'm pretty sure she does not live is a vacuum!
Sue has been an audiophile for longer than I have, and has owned and auditioned a lot more equipment than most people on AudiogoN. Should she not have an idea what to expect when she auditions a pair of amps from a company that has been around for a long time and has a well established reputation. I don't know if Susie liked the amps or not, I would guess she isn't selling her currrent system to buy them! BUT that's not the issue.
Who goes shopping, or auditioning without a good idea of what they are getting into? That person would be foolish! At that point in the process a person should have a good idea of what to expect.
The blindfold is not the issue, you are questioning the integrity of the reviewer! If you want to play that game, you are certainly welcome but don't cloak it under the mantle of objectivity. Either something sounds good, or it doesn't, it doesn't matter what headgear is involved...
Correct, however, referring to my example, I will never know if these amps really sound good or not because SK is NOT giving an objective observation. Her predjudices and impartiality are clouding the review. Period. How can anyone dispute this????? She even comes right out and tells you.
I'm not questioning her integrity. She probably has lots. Or maybe none. I don't know and I don't purport to.
I'm commenting on the process which most reviewers are subject to.
So should magazines hire writers who have no preconceived notions about a product? Do you want to read a review written by someone who does indeed live in a vacuum???
How can someone without years of listening experience write a review??? You have sunk below the level of absurd...
Nrchy; You're missing the point. Sue can continue to review. Just don't tell her what's she's listening to. That's not absurd at all. Just say, Sue, you'll be listening to a solid state amp in your system. Tell us what you think.
I'm not sure how we keep missing each other on this. Doesn't this make sense?
I don't think you understand the process involved in working for a magazine, or reviewing a piece of gear...
The reviewer lives alone, and has to take possession of the amps, get them into place, hook them up, andthen conduct a review. It is impossible for her not to know what is being reviewed. Reviewers don't go off to some magic island named vacuum and sit there for a few weeks to do their job. This is a part-time, supposedly fun (if you call getting reamed out by every audiophile who disagrees, fun) job for people who are otherwise gainfully employed.
a reviewer should be objective when appraising the sonic merits of a component. a reviewer should be an accurate reporter of what he/she hears--no more no less.
it is counter productive for the reviewer to express a sentiment about the component reviewed. just report the facts and leave the biases and persuasion out of the review.
Magic Island named vacuum, can I get directions to this place? Nrchy Thank you!! You added a nice laugh to an otherwise long day! I still do want directions though, as I am a tube guy.LOL
As a tube guy I don't think you would be welcome on the magic island of Vacuum, you're living in the past!!! : )
MrTennis, People want reviews to be inviting, and so reviewers offer personnal anecdotes to pull the reader in, this is all IMNSHO Ms Kraft did in her review. I could be wrong, since I have not read the review, but I think the attacks on the writer are unwarranted. No one knows the experience or intention of the writer except the writer. To subject her to your prejudices is the worst type of STEREOtype... and to remove all value from this or any other review.
I'm surprised by your question and your assertions.
You should know that forced choice tests are the only scientifically valid kind of tests, even though it may be impractical for evaluating audio equipment. If you can't reliably distinguish among two or more conditions (amps, power cords, or whatever), I have no interest in hearing your subjective chat about how superior one is over another. But high-end audio is so full of mysticism the last thing that is wanted is objective evaluation. The emperor may in fact have no clothes.
So, yes, blind tests can be useful, but difficult to implement. I suspect most manufacturers do such testing to evaluate proposed improvements in their own products and to compare their products with a competitor's. I've certainly spent my time in labs doing blind listening tests -- worst of all is testing in a deep LF anechoic chamber, very uncomfortable feeling.
personal anecdotes are ancillary to the review. all of the preliminary repartee has no bearing on the review.
in some of my articles, i use poetry and engage in expostulary philosophical remarks. my reviews are dsiapassionate. i try not to influence the reader.
you may read my reviews on audiophilia.com . currently you can peruse a write-up of the 2007 ces show.
You must have a verified phone number and physical address in order to post in the Audiogon Forums. Please return to Audiogon.com and complete this step. If you have any questions please contact Support.
