Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera


Folks-

Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.

In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.

As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:

1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;

2.  Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars; 

3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?

I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?

Thanks for listening to my ranting...Gerry 
128x128Ag insider logo xs@2xgerryah930
and where is the objectivity in this domain?

Good question. There's little to none. My conclusion is if it's not blind, they didn't hear it. The only time I would pay any attention whatsoever to a subjective review is if that person continually came to the same results as I.

Regarding #2, you'll usually see an influx of items for sale when the next "hot" thing comes out. The herd is always chasing the newest rabbit. Doesn't necessarily mean people are dumping them for cause.

Regarding #3, there are plenty of technical articles showing why MQA is just a money-making scheme.

BTW, there have been blind tests on SACD, including an AES paper. I think you can guess what the results were. However, there was an audible difference in the noise floor when there was no signal and high system gain at a level where music would have been "unpleasantly loud".
I had a job trading bits for pieces
We'd make wrinkles, advertise them as creases
After reading both regularly for about three years now, my feeling is that Stereophile is more objective and professional in reviewing equipment than The Absolute Sound. Although the latter magazine occasionally publishes well-written reviews, sometimes it is impossible to tell whether the price quoted for a speaker is for one speaker or a pair. Stereophile is not uniformly positive, and points out when the item being reviewed doesn't work, or stops working (or blows up), and when it provides no discernible improvement in sound quality. They also give the manufacturer a forum to respond to reviews.

I have found that both magazines have helped me get educated on new stuff in the hobby after a 20-year hiatus. And in my opinion the best writers in the audiophile universe (Art Dudley, Herb Reichert and Michael Fremer) are all at Stereophile.
I put in my vote for objective and professional reviewer REG of TAS. Everyone in the hobby should check out the articles at regonaudio.com
+1 john_g, my feelings exactly. I was poised to print the same post until I read yours.

Tom

ps- don't forget Sam Tellig (Tom Gillette). He was my hands-down favorite, unfortunately he no longer writes.