Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera


Folks-

Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.

In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.

As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:

1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;

2.  Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars; 

3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?

I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?

Thanks for listening to my ranting...Gerry 
128x128Ag insider logo xs@2xgerryah930
The main purpose of most reviewers is self preservation. They pander to certain manufactures and favour certain flavour of the month products. Honesty is so far down the list of priorities it's seen as a mythical quality to all but the gullible. On a sinking ship integrity, because of its weight, is the first thing that gets thrown overboard.

Magazine reviewers are almost entirely misinformed and misguided idiots to a man. Its a bit like selling soap, every year new improved products are launched...
Yet somehow the reference products of yore remain unsurpassed.

For decades they have lied about digital sources, they didn't have a clue about speaker isolation, recommending spiking ad infinitum, and most of the hacks can't write for toffee. Memorable reviews? Ha! You'd be lucky to find a dozen.

Hi-Fi journalism is akin to mainstream pornography. It can be easy on the eye, but bears little relationship to reality. Trust it at your peril. Now if only the manufacturers weren't such liars. Now if only we didn't like the lies so much....


cd318
The main purpose of most reviewers is self preservation. They pander ... Honesty is so far down the list of priorities ... integrity ... is the first thing that gets thrown overboard.

Magazine reviewers are ... misguided idiots to a man ... For decades they have lied ...
Hi-Fi journalism is akin to mainstream pornography ... if only the manufacturers weren’t such liars ...
Why so angry? What makes you such an objective observer of all things audio?
It is gradually becoming increasingly apparent that user "cd318" has his own agenda that he is enforcing at the same time that he reserves the right to himself to attack others who disagree with him however a major difference is that we don't know what users' "cd318" interest is in the matter which makes it difficult/impossible for readers of this forum to meaningfully assess  his "opinions" on matters that he states here with such forcefullness. But I admit comparing Hi-Fi news to pornography is a little bit weird and makes me wonder what he is doing with his Hi-Fi News magazines but of course that is his solitary business and not for us to question challenge criticize or condemn so long as no one is being hurt it is fine.
" Honesty is so far down the list of priorities it's seen as a mythical quality to all but the gullible. On a sinking ship integrity, because of its weight, is the first thing that gets thrown overboard. " -  cd318

" Hi-Fi journalism is akin to mainstream pornography. It can be easy on the eye, but bears little relationship to reality " -cd318

Spot on and hilarious - one of the best posts I've seen on this forum so far. The recent, sustained push for MQA in the "audiophile press" has been a great example of how far they're willing to go to support a farce aimed at emptying the wallets of the gullible. It's a clear window into the lack of "journalistic integrity" that exists.
cj1965
The recent, sustained push for MQA in the "audiophile press" has been a great example of how far they're willing to go to support a farce aimed at emptying the wallets of the gullible.
Do you have any examples of this "sustained push?" Without examples, it's difficult to know what you're talking about.