2. cables
  3. 1118611066

Reviews with all double blind testing?

In the July, 2005 issue of Stereophile, John Atkinson discusses his debate with Arnold Krueger, who Atkinson suggest fundamentally wants only double blind testing of all products in the name of science. Atkinson goes on to discuss his early advocacy of such methodology and his realization that the conclusion that all amps sound the same, as the result of such testing, proved incorrect in the long run. Atkinson’s double blind test involved listening to three amps, so it apparently was not the typical different or the same comparison advocated by those advocating blind testing.

I have been party to three blind testings and several “shootouts,” which were not blind tests and thus resulted in each component having advocates as everyone knew which was playing. None of these ever resulted in a consensus. Two of the three db tests were same or different comparisons. Neither of these resulted in a conclusion that people could consistently hear a difference. One was a comparison of about six preamps. Here there was a substantial consensus that the Bozak preamp surpassed more expensive preamps with many designers of those preamps involved in the listening. In both cases there were individuals that were at odds with the overall conclusion, and in no case were those involved a random sample. In all cases there were no more than 25 people involved.

I have never heard of an instance where “same versus different” methodology ever concluded that there was a difference, but apparently comparisons of multiple amps and preamps, etc. can result in one being generally preferred. I suspect, however, that those advocating db, mean only “same versus different” methodology. Do the advocates of db really expect that the outcome will always be that people can hear no difference? If so, is it the conclusion that underlies their advocacy rather than the supposedly scientific basis for db? Some advocates claim that were there a db test that found people capable of hearing a difference that they would no longer be critical, but is this sincere?

Atkinson puts it in terms of the double blind test advocates want to be right rather than happy, while their opponents would rather be happy than right.

Tests of statistical significance also get involved here as some people can hear a difference, but if they are insufficient in number to achieve statistical significance, then proponents say we must accept the null hypothesis that there is no audible difference. This is all invalid as the samples are never random samples and seldom, if ever, of a substantial size. Since the tests only apply to random samples and statistical significance is greatly enhanced with large samples, nothing in the typical db test works to yield the result that people can hear a difference. This would suggest that the conclusion and not the methodology or a commitment to “science” is the real purpose.

Without db testing, the advocates suggest those who hear a difference are deluding themselves, the placebo effect. But were we to use db but other than the same/different technique and people consistently choose the same component, would we not conclude that they are not delusional? This would test another hypothesis that some can hear better.

I am probably like most subjectivists, as I really do not care what the outcomes of db testing might be. I buy components that I can afford and that satisfy my ears as realistic. Certainly some products satisfy the ears of more people, and sometimes these are not the positively reviewed or heavily advertised products. Again it strikes me, at least, that this should not happen in the world that the objectivists see. They see the world as full of greedy charlatans who use advertising to sell expensive items which are no better than much cheaper ones.

Since my occupation is as a professor and scientist, some among the advocates of double blind might question my commitment to science. My experience with same/different double blind experiments suggest to me a flawed methodology. A double blind multiple component design, especially with a hypothesis that some people are better able to hear a difference, would be more pleasing to me, but even here, I do not think anyone would buy on the basis of such experiments.

To use Atkinson’s phrase, I am generally happy and don’t care if the objectivists think I am right. I suspect they have to have all of us say they are right before they can be happy. Well tough luck, guys. I cannot imagine anything more boring than consistent findings of no difference among wires and components, when I know that to be untrue. Oh, and I have ordered additional Intelligent Chips. My, I am a delusional fool!
tbg
pabelson
344 posts
 
Yes, beauty can grow on you. But notice that it's not the lady who's changing. It's you. What does that tell us about long-term comparisons?

TBG: Yes, there are far too few objectivists to make a market. That's why the largest-selling magazine in the US that reviews audio equipment is that subjectivist redoubt . . . Sound & Vision.
shadorne's avatar
Ag insider logo xs@2x
shadorne
7,477 posts
 
Pabelson,

Thanks for your posts. Although I have never done blind tests, my experience with audio equipment matches the DBT reports perfectly. I simply can't detect audible differences between different cables, CD players and solid state amps provided, of course, they are of a minimum quality.

I do, however, detect large audible differences between different speakers, their placement and the room in which they are played. I also notice large differences in low powered SS amps versus high powered SS amps but only when driven at a high demanding levels.

On my Anthem AVM 20 preamp/processor I am able to play music either direct anaolg or through the A to D and then D to A circuit....a kind of test....again I hear no difference as long as all the tone controls and settings are the same. So I typically use the A to D and D to A in the circuit becuase it allows me to use base management for my subwoofer...something that does need adjustment (for room) and makes a tremendous difference when adjusted to my taste (digital filters providing great flexibilty). Furthermore, I can use the digital out or the analog out from my various CD players and again I hear no audible difference provided tone control/volume is the same...although I will admit that the digital output can have less background hiss S/N at extremely high levels, but this is not always audible either (setup dependent).

What does this mean for me when buying equipent?

1) Choose CD's and SS Amps for their features (speaker protection, tone controls, souund processing capability, warrenty etc.) For example, I like CD changers as I don't have to mess with CD cases. As for amps, I prefer overly powerful SS amps with significant headroom in order to minimize distortion from clipping.
2) Spend most or your money on the speakers, as this is the single biggest variable and adds the most distortion in the whole setup.
3) Buy good quality (shielded) but not exhorbitantly priced cables and inteconnects.
4) Pay attention to which room is used for playback and what are the wall/floor coverings and speaker placement.
puremusic
226 posts
 
With apologies to Shakespeare and all logicians:
"To DBT or not to DBT is or is not the question."

Hi Pabelson,

Your quote points us to the central point of this discussion:
"Yes, beauty can grow on you. But notice that it's not the lady who's changing. It's you. What does that tell us about long-term comparisons?"

It tells us what neuroscience has discovered. The brain is much more plastic than once believed. It is not static like electronic circuits. The brain circuitry and its chemistry change. New interneuronal connections are formed and concentrations of neurotransmitters and other brain chemicals change. So, what the brain could not distinguish one day, it may LEARN to distinguish in subsequent exposures to the experience. We have experienced this learning phenomenon as students, as professors, and as audiophiles. This is part of our growth and evolution. A double-blind test based on short-term listening sessions may not allow enough time for the brain circuitry and chemistry to reconfigure itself to discern the difference. Therefore, if a short-term double-blind test does not show a difference between two amps, it would not be correct to conclude that there was no difference between the amps, only that that particular test did not reveal a statistically significant difference. A double-blind test showing a positive difference may be useful for audiophiles, while the test showing no difference is an inconclusive statement about the amps.

Incorrect interpretations can also be made for long-term double-blind tests. History of science shows us that even the hard sciences like physics are not immune from making incorrect interpretations. A committment to truth and critical thinking helps purify science to better the human condition. Otherwise, our implicit assumptions may yield tautological statements similar to the very first statement in this post. Although it is logically valid, it does not contain useful information for the audiophiles.

Best Regards,
John
gregadd
574 posts
 
my new policy on audiogon is to post my opinion and let it stand on it's own merit. I no longer feel the need to respond to every competing opinion. I'll let the readers make thier own conclusion. I do however reserve the right to respond crticism directed at me.

My approval of DBT is in no way an endorsement of the ABX test.

Just because you beleive in DBT or ABX testing does not make you an objectiive. DBT proponents have yet to show me where they have used it to advance the state of the art. They are as biased as anyone else. In fact the inventer of the ABX gave this as his reason for inventing the ABX box-he was upset that audio companies could be destroyed by audio reviewers who did not know what they are talking about. Thus DBT/ABX was invented to attack the integrity of audio reviewers. Not as an objective scientific tool.

The intial tests where short term on inexperienced listeners. That is a fact. To furhter demonstrate thier lack of objectivity it is the proponents of DBT/ABX, when confronted with the fact that reviewers like Michael Fremer were in fact able to match A&B to X, attacked the validity of thier own test. In effect they concluded that because they knew there was no difference between amps he must be using some trick formulated by his knowledge of the amplifiers under test.
No one argues that amps and cables sound the same. Nothing is further from the truth. That is exactly what they argue calling it snake oil and making vial insults to those who design, sell, buy and review it.

Feel free to remain wedded to frequency response, distortion figures and output impedance if you like. You don't need a blind test for that because it is so easily measured. You may clean your palette with the occaisional blind test. Ultimately you are going to have to listen. This is what all the manufactures of good equipment do.
pabelson
344 posts
 
Let's not be so selective about what neuroscience has been discovered, John. It, along with psychoacoustics, has indeed discovered that it can take time to learn the sound of something, and the difference in sound between things. But they've also discovered that, once you've learned those differences, the best way to confirm that those differences are really there is through short-term listening tests that allow you to switch quickly between the two components. So why is it that a reviewer, who supposedly spends weeks "getting to know" a component, and who also owns a reference component which he also knows well, can't hear a difference between the two in such a test?

My point about how we change was aimed at the reviewer who reports differences between the component under review and something else he may have heard months before, but doesn't have now. He's claiming to do something that your neuroscientist/psychoacoustician has found to be impossible.
More to discover