2. cables
  3. 1118611066

Reviews with all double blind testing?

In the July, 2005 issue of Stereophile, John Atkinson discusses his debate with Arnold Krueger, who Atkinson suggest fundamentally wants only double blind testing of all products in the name of science. Atkinson goes on to discuss his early advocacy of such methodology and his realization that the conclusion that all amps sound the same, as the result of such testing, proved incorrect in the long run. Atkinson’s double blind test involved listening to three amps, so it apparently was not the typical different or the same comparison advocated by those advocating blind testing.

I have been party to three blind testings and several “shootouts,” which were not blind tests and thus resulted in each component having advocates as everyone knew which was playing. None of these ever resulted in a consensus. Two of the three db tests were same or different comparisons. Neither of these resulted in a conclusion that people could consistently hear a difference. One was a comparison of about six preamps. Here there was a substantial consensus that the Bozak preamp surpassed more expensive preamps with many designers of those preamps involved in the listening. In both cases there were individuals that were at odds with the overall conclusion, and in no case were those involved a random sample. In all cases there were no more than 25 people involved.

I have never heard of an instance where “same versus different” methodology ever concluded that there was a difference, but apparently comparisons of multiple amps and preamps, etc. can result in one being generally preferred. I suspect, however, that those advocating db, mean only “same versus different” methodology. Do the advocates of db really expect that the outcome will always be that people can hear no difference? If so, is it the conclusion that underlies their advocacy rather than the supposedly scientific basis for db? Some advocates claim that were there a db test that found people capable of hearing a difference that they would no longer be critical, but is this sincere?

Atkinson puts it in terms of the double blind test advocates want to be right rather than happy, while their opponents would rather be happy than right.

Tests of statistical significance also get involved here as some people can hear a difference, but if they are insufficient in number to achieve statistical significance, then proponents say we must accept the null hypothesis that there is no audible difference. This is all invalid as the samples are never random samples and seldom, if ever, of a substantial size. Since the tests only apply to random samples and statistical significance is greatly enhanced with large samples, nothing in the typical db test works to yield the result that people can hear a difference. This would suggest that the conclusion and not the methodology or a commitment to “science” is the real purpose.

Without db testing, the advocates suggest those who hear a difference are deluding themselves, the placebo effect. But were we to use db but other than the same/different technique and people consistently choose the same component, would we not conclude that they are not delusional? This would test another hypothesis that some can hear better.

I am probably like most subjectivists, as I really do not care what the outcomes of db testing might be. I buy components that I can afford and that satisfy my ears as realistic. Certainly some products satisfy the ears of more people, and sometimes these are not the positively reviewed or heavily advertised products. Again it strikes me, at least, that this should not happen in the world that the objectivists see. They see the world as full of greedy charlatans who use advertising to sell expensive items which are no better than much cheaper ones.

Since my occupation is as a professor and scientist, some among the advocates of double blind might question my commitment to science. My experience with same/different double blind experiments suggest to me a flawed methodology. A double blind multiple component design, especially with a hypothesis that some people are better able to hear a difference, would be more pleasing to me, but even here, I do not think anyone would buy on the basis of such experiments.

To use Atkinson’s phrase, I am generally happy and don’t care if the objectivists think I am right. I suspect they have to have all of us say they are right before they can be happy. Well tough luck, guys. I cannot imagine anything more boring than consistent findings of no difference among wires and components, when I know that to be untrue. Oh, and I have ordered additional Intelligent Chips. My, I am a delusional fool!
tbg
henryhk
648 posts
 
Put it another way, double blind testing cannot work IMO in audio precisely because there is no "absolute" definition on "what sounds better" to begin with (with apologies to HP). How do we define it the first place? Measurements are fine but in only in terms of allowing us figure out what aspects of sound (jitter, frequency range, channel separation etc etc)that contribute to what we are hearing...but in the end its a VALUE judgement. Some like tubes, some like SS. Some need gigantic floorstanders for frequency extension, some prefer mini-monitor accuracy & imaging. Some prefer transparecy, air etc others image solidity, warmth etc. The evaluators who comment after a DBT are including their own subjective value judgements as well. You cannot answer a question that is ill defined in the first place. In this light someone else perhaps half joking said what about cars, wine, literature whatever....and in a way he is right on the money: in all of these pursuits value judgements occur.

If we are to just focus aspects of sound and have no value judgement...i.e just want to know say the level of jitter of one against the other etc...then we can just measure for the most part, no need for DBT again.

The one aspect I think DBT can be used perhaps is not comparing one brand vs another, but within brand and the same model: for example you have the exact same system set up, and then test a new model: one has more jitter than the other...then one has upsampling switched on, the other doesn't etc etc...by the manufacturers....this way one may try to investigate what matters most, a priority schematic, if you will one, to audiophiles or the public at large and then devise products or even array of options on the same product to maximize revenue or provide tailor-made solutions for various segements. Obviously this is more of marketing strategy than an answer to the "holy grail".
henryhk
648 posts
 
Sooner or later someone is gonna start advocating db testing for cars...yikes!
pabelson
344 posts
 
It is a well structured experiment that differs greatly from what we normally hear and how we hear it.

This is just an astounding statement. How in the world can simply not telling someone what they are listening to affect what they *hear*? I'll grant you, it can certainly affect what they think about what they hear, but that's just the point. What they think is a function of things besides what they hear, and DBTs isolate the non-sonic effects.

Note that there's no necessary contradiction between these two statements:
1) Harry can't hear a difference between A and B.
2) Harry prefers A to B.
Both can be true. All it means is that Harry prefers A to B for some reason other than its sound (even if he thinks the sound is the reason).
pabelson
344 posts
 
Greg: Your generally thoughtful and balanced letter was, in my opinion, a little too balanced. Here's where you went astray:

The proponents of dbt...want to engage in very short tests conducted by the uninitiated. Most proponents of dbt use it to try and prove what they already have concluded.eg cables and amps all sound the same.

This reflects a basic misunderstanding. Objectivists don't want short tests, we want good tests. (All the research suggests that short tests are in fact better tests, but DBTs can be any duration you want). And a requirement of good DBTs is that you provide the subjects with adequate "training," meaning that they are familiar with the sound of the equipment they are comparing. The "uninitiated" make very poor test subjects.

Finally, no one argues that all cables and amps sound the same, and that's not the purpose of DBTs. The purpose of DBTs is to determine *which* components sound the same, and which do not.

How about a dbt between vinyl and digital. Or electrostatic and dynamic speakers- tubes and solid state.

All of these have been done, at one time or another. Vinyl and digital are easily distinguishable--unless the digital is a direct copy of the vinyl. Speakers are always distinguishable in DBTs. Tube and solid state amps are often but not always distinguishable. When they are distinguishable, it's usually because the tube amp is underpowered and clipping (though very mellifluously, as tubes are wont to do!), or because the output impedance of the amp is interacting with cable and speaker to produce frequency response errors.
qualia8
49 posts
 
WARNING: LONG POST -- LIFE HISTORY AND ITS ILLUSTRATION OF BIASES -- YOU MAY WANT TO SKIP

I went through grad school with a $150 boombox. As a classical music lover, I obviously wasn't happy with it, but what was I going to do? Sell my '87 Buick and walk? Audio was not *that* important to me. So it wasn't until I got a job that I decided to invest a little something in a decent 'stereo'. Still, I was married and my wife was in law school, racking up debts. Not knowing anything about hifi, I decided to get a simple home theatre setup. I went to Best Buy, dropped three hundred on a Yamaha receiver, and another couple hundred on a 5.1 speaker set-up. ($100 off b/c I bought the two together.) Some cheap cables, and I was headed home to set up my new rig. Hooray! And man, this thing came with a sub!

You know what happened, of course. The system actually did pretty well with movies. I don't care all that much about HT being perfect. Eminem's 'Eight Mile' was the first movie I watched with the new setup, and it ROCKED! Highs were crystal clear. Very sharp. And the bass, or actually, it was mid-bass, b/c that sub doesn't go too low, was nice and full in my apartment. Gladiator was great too. Cool.

Then I popped in my cds. I wanted *so* badly to like what I heard. After all, my wife was already pissed that I had spent $500. "$500? And you don't like it? What's wrong with you? If you're going to be so picky, you should have gone to law school rather than taking forever to write a dissertation." (I still wasn't done at the time.) She didn't even know about the extra 100 I had spent on a Monster surge protector and cables.

But it sounded terrible. Mid-range just sucked. There's no other way to say it. And treble portions were highly highly annoying.

I stuck with the system for the next year or so. After a separation from my wife, I did what any lover of sound would do, and finally allowed myself in the local hifi shop. (It was only two blocks from my apartment.) I walked in with the idea of purchasing new monitors for the fronts and leaving the rest of the 5.1 system in place. Explaning my situation, the staff (quite helpful, really) suggested Paradigm monitor bookshelves. They were a few hundred bucks and sounded great in the store. There it was -- lifelike voices, not the tinny, metalic sounds I heard at home. Ahhhh!!!

Before I left with the Monitors, one of the sales guys said he had a pair of Studio v.3's I should listen to before making a purchase. Well... I listened, and *wow*. Incredibly accurate sound. It was nothing you could hear with any combination of Best Buy equipment. I bought the speakers on a pretty hefty discount, with stands, and charged home to listen.

The improvement *was* dramatic, don't get me wrong, but still not anything like I heard in the store. Hmmm... Could it be the other stuff in my system? Nah... Cd players are all the same. And amps too. The Yamaha was rated *way* above the requirements for my new Paradigms. And so what if my source was an old dual VHS/dvd player? Bits is bits, right? So it must be my room.

I spent another several months trying to like the sound. Very quickly, I discovered that speaker positioning mattered, and room treatment too. I made a lot of adjustments, but my sound was never *smooth*, as it was in the store. Hmmm...

About this time, I started researching audio. I was relieved to find out people liked my Paradigms for a "budget" speaker. "Budget? Seriously?" I thought. But everyone seemed to think that source and amplification were also important. And there was this thing called a "preamp".

I went back to the audio store and tried some better receivers -- Pioneers with room eq -- but the sound still wasn't to my liking. Sure it was loud, dynamic, and even full. But it left me cold. I pointed to some shiny gear across the room. "What about that?"

"Oh, you don't want that. It's just two-channel. You want home theatre, right?"

"Well yeah, but first and foremost, I want something that sounds good."

So he played me a Musical Fidelity integrated and cd player (around $1,500 each), with my Paradigms. Unbelievable. I just sat and listened for about two hours, entranced, letting the music work its magic on me.

I couldn't afford the MF, but there was a demo Classe, which sounded very similar in the store, only significantly less cash. I brought that home and auditioned it. Definite improvement on the Yamaha, or so I thought.

I bought it and sold the sub + sats on Ebay. Now I was *there*, right? No. I still got annoyed. But closer. Definitely closer.

Anyway, about this time, I discovered Audiogon. I also started talking to my brother-in-law, who had tried dozens and dozens of combinations of amps and speakers to get vocal music right. I realized I was only at the beginning. I was just started on the audio path. Damn. I thought I could just walk into Best Buy, walk out, and be done with it. I had no idea this would be a hobby, and a long-lasting costly hobby at that.

Anyway, I still have the Classe. And now I wonder whether it actually sounds any better than my old Yamaha. Even if it doesn't, objectively speaking, I think it does, subjectively. Because it's a really pretty amp. It has this super-heavy milled steel remote and the display, volume knob, and everything, just ooze quality. (Ok, the outputs don't. They seem cheap.) I can't help but look at my setup when I listen, and I much prefer looking at the Classe.

Maria Callas had a magnificent voice, but she was also hot, and I'm sure that added to the experience of opera-goers of the time. Speaking for myself, I prefer a grotesquely fat and ugly soprano who sounds good to a waifish beauty who sounds strained, BUT, other things being equal, a beautiful soprano actually *sounds* better in the typical soprano role. I once saw Angelina Gheorghiu in the role of Mikaela in Carmen at the Met. Gorgeous coloratura soprano, but also, she was beautiful, at least from the cheap seats where I sit. Took the breath out of my chest. I bet Gheorghiu wouldn't prove that much better than her fatter and uglier peers in blind comparison. But at the opera, you ain't blindfolded.

Maybe what happened when I looked across that showroom and spotted the shiny MF gear was just love. Just as hunger is the best sauce, love makes things sound better. A *lot* better.
More to discover