The two "Peters" (Aczel and Moncrieff) made a determined attempt to avoid "favoritism" and "commercialism". Check out The Audio Critic (TAC) and International Audio Review (IAR). Listening combined with measuring for honest and forthright assessment of gear! Nothing since has been at the same level (though Brit publication Hi Fi News does try for " objectivity " - but is still bound to its advertising base).
Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera
Folks-
Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.
In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.
As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:
1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;
2. Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars;
3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?
Thanks for listening to my ranting...Gerry
Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.
In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.
As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:
1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;
2. Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars;
3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?
Thanks for listening to my ranting...Gerry
- ...
- 64 posts total
roberjerman The two "Peters" (Aczel and Moncrieff) made a determined attempt to avoid "favoritism" and "commercialism". Check out The Audio Critic (TAC) and International Audio Review (IAR). Listening combined with measuring for honest and forthright assessment of gear! Nothing since has been at the same level (though Brit publication Hi Fi News does try for " objectivity " - but is still bound to its advertising base). There is a fine line between “honest and forthright assessment” and closed mindedness and self aggrandizement. “What I have learned after six decades in audio.” - Peter Aczel Oh, please! |
willemj If the methodological standards common in audio publications and audiophile forums applied to medical science life expectancy would be halved.That's a false comparison. Audio publications are hobbyist magazines, not scientific journals. And there's plenty of nonsense in medical science, too. |
@gerryah930 - have you considered that products that are reviewed are actually pretty darn good? And perhaps - that's why the review is (almost) always favorable? After reading lots of reviews, I have found the I can generally tell when a product may be falling a little short of the reviewers expectation - they "generally' start to nit-pick. And the opposite occurs also - if the product is fantastic the reviewer goes out of their war to extol the virtues of a component. Sometimes "numbers" are not enough - you just have to learn to "read between the lines". As for Diana Krall... - a great many of her albums are a product of exemplary sound engineering/recording and final mixing and more often than not lacks that "wall of sound", which allows a reviewer to focus on the details in the recording. - As opposed to another delightful singer, Norah Jones, who's albums are also very well engineered but are quite varied in their production and can sometimes lack the more natural tones of a Diana Krall album. These observations are result of hundreds of hours of listening I my self have conducted whilst reviewing (as an amateur) a small number of products. Unlike many, my reviews were not paid for, but I was enthused by the outstanding abilities of those products. I'm not defending professional reviewers - but there is always another side to the coin Having experienced just how difficult it is to write a review that is engaging, informative, concise, insightful and more importantly useful to the reader, I think it is important for people to consider some of the "intangibles" at play. BTW- do you collect stats on how many products are returned without a review? Regards - Steve |
- 64 posts total