Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera


Folks-

Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.

In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.

As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:

1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;

2.  Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars; 

3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?

I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?

Thanks for listening to my ranting...Gerry 
128x128Ag insider logo xs@2xgerryah930

browndt
No negative reviews, in my view, means no other review can be trusted. I know collusion when I see it and that is exactly what stereo magazines and manufactures have done for years. Any objective scientific review will have negatives. I should know I have edited scientific publications for years, where lots of critique can be found.

>>>>Actually, many if not most reviews have always included negative aspects, if any, as well as positive ones of the device under test. For example frequency response curves and radiation patterns, distortions of various types. The idea they would Gild the Lilly 🌷doesn’t really hold water IMHO. In fact, one reason they provide negative aspects is too prevent or dispel the impression that the reviewer or magazine is biased or that there is any sort of collusion. They aren’t stupid.
Good read, so far....and that despite the Cable Drifting : )

I'll add to @williewonka 's point:

...have you considered that products that are reviewed are actually pretty darn good...

In my view, most gear that makes it to market is very good, especially when considered on it's own. That in itself eliminates highly 'negative' reviews. 

I'm searching my memory for a component that I've had in system that I would consider a turd on arrival and which remained so.....

This includes speakers which are quite variable (as you know) and some  tube components (early in my journey) that were overly euphonic. For both component classes it was not that they were poor performing but rather that I didn't care for their sound.

For me, reviewers offer a great service that I am grateful for regardless of bias or vested interests. It's the way of the world, even the medical and science ones. Reviewers I have personally interacted with have been no different than my audio friends and acquaintances. A pleasure.

I can also understand, if that's how one chooses to roll, ignoring their reviews. I, however, find the toxic mudslinging unseemly. 
Reviews for audio anything should be taken with a grain of salt. It is up to the individual to realize that a certain amount of product bias exists in all reviewers skewed toward their benefactors or personal biases. Their publication can be useful for a buyer to determine which direction he may want to try but should never be used as the gospel truth. Audio is a very personal matter and a 100000 dollar system is no better suited to propel the individual human into audio nirvana than a 1000 dollar system. The human condition determines when nirvana has been obtained, not reviews. All of us in these forums have reached nirvana at some time of our continuing search for a better nirvana without reviews simply by trial and error. We still do it. Just look at all the equipment ads. We are our own very best reviewers. All different and yet the same. One persons failure to enter their nirvana opens the gates to nirvana for someone else.
Isn't it wonderful? 

@browndt Absolutely!  There are numerous manufacturers that for one reason or another are not members of "the club".  They good ones soldier on, on the strength of their product, word of mouth, etc., without a boost from S-phile or AS.