Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera


Folks-

Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.

In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.

As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:

1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;

2.  Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars; 

3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?

I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?

Thanks for listening to my ranting...Gerry 
128x128Ag insider logo xs@2xgerryah930
" Honesty is so far down the list of priorities it's seen as a mythical quality to all but the gullible. On a sinking ship integrity, because of its weight, is the first thing that gets thrown overboard. " -  cd318

" Hi-Fi journalism is akin to mainstream pornography. It can be easy on the eye, but bears little relationship to reality " -cd318

Spot on and hilarious - one of the best posts I've seen on this forum so far. The recent, sustained push for MQA in the "audiophile press" has been a great example of how far they're willing to go to support a farce aimed at emptying the wallets of the gullible. It's a clear window into the lack of "journalistic integrity" that exists.
cj1965
The recent, sustained push for MQA in the "audiophile press" has been a great example of how far they're willing to go to support a farce aimed at emptying the wallets of the gullible.
Do you have any examples of this "sustained push?" Without examples, it's difficult to know what you're talking about.

You only need to look at the last few months' issues of Stereophile and Absolute Sound to see a multi-pronged and significant push for MQA.
twoleftears"You only need to look at the last few months' issues of Stereophile and Absolute Sound to see a multi-pronged and significant push for MQA."

I do not get those publications I do not believe in my country so it would be helpfull if some of those here who criticize this "push" would provide some examples of what they find objectionable because otherwise it is hard to follow your line of reasoning about the audiophile press.
It certainly applies to audio press in my country (the Netherlands) that there is a push for MQA. That push does not come with any proper explanation of what it is, or how it is  better.
Well I just think it is funny that so many here seem to complain about the "big push" for MQA by the audiophile press but they give no examples of specific articles or authors but i guess it is like some of the other matters here on Audiogon where people make claims and then expect others to proof they're claims for them!

I think it is also funny that people are complaining about what the audiophile press is writing about because after all they are magazines for audiophiles what do you want them to write about? Maybe what you want is for them to get into the topics you explore here like telling others' that they are deluded or snakeoil salesmen or matters of politics and economics would that be better for you??
I did A/B on MQA Tidal stream vs. vinyl rip 24/96 from digitally mastered Adrean Belew "Mr Music Head"
MQA 24/96 stream was substantially inferior to vinyl rip.

" Well I just think it is funny that so many here seem to complain about the "big push" for MQA by the audiophile press but they give no examples of specific articles or authors but i guess it is like some of the other matters here on Audiogon where people make claims and then expect others to proof they're claims for them! " - Clearthink

Oh puhleeeeeze. Spend five minutes of your time Googling Stereophile and MQA and you'll have all the proof of a "push" that you need.  Nobody is going to waste time giving you proof of the obvious. If you're that lazy and you don't feel like looking for yourself at what most everyone here has seen and talked about - don't expect any of us to serve it up to you on a silver platter. It's what "objectivists " refer to as "common knowledge".
It is often difficult to decipher a Stereophile review. As someone above noted, they
may damn a product with faint  praise.  Many of their reviews seem inconclusive or unenthusiastic - which usually means "there are better products out there".  When the review for a $25,000 amp makes it seem like it's merely adequate, something is not quite right.  They do seem to go out of their way to avoid conclusory judgments.  
 
The text may give some praise to satisfy the advertiser, but the important part is the measurements section. Those measurements may tell a rather different story. Sometimes, when the discrepancy is too obvious, John Atkinson will mention this, but at other times you really have to do the interpretation for yourself. It is not hard.
What is also obvious is that many of these pieces of expensive audiophile electronics perform far worse than much cheaper electronics from mainstream manufacturers. To find measurements for those you usually have to look elsewehere, since Stereophile does not normally test such stuff. But Google is your friend. See here for an example: http://www.avhub.com.au/product-reviews/hi-fi/yamaha-a-s500-amplifier-review-test-395710

cj1965
Oh puhleeeeeze. Spend five minutes of your time Googling Stereophile and MQA and you'll have all the proof of a "push" that you need. Nobody is going to waste time giving you proof of the obvious. If you're that lazy and you don't feel like looking for yourself at what most everyone here has seen and talked about - don't expect any of us to serve it up to you on a silver platter. It's what "objectivists " refer to as "common knowledge".

That is so funny first you so-called "objectivists" say that we have to proof what you think because what we think is so unusual now you say we have to proof what you think because it is so obvious you want it both ways! Why not just say "I am an objectivist I am a scientist I know what I am talking about you are lazy go find you're own proof for everything." That is just crazy.
The text may give some praise to satisfy the advertiser, but the important part is the measurements section.


Hahahaha, that is hilarious!
Thanks for my daily chuckle. ;^)