Gerry,
Drive the car. Listen to the gear. It's all personal.
Kenny
Drive the car. Listen to the gear. It's all personal.
Kenny
Publication bias and confounders in product reviews - TAS, Stereophile, Audiogon, etcetera
Folks-
Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.
In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.
As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:
1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;
2. Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars;
3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?
Thanks for listening to my ranting...Gerry
Since I am a research professor at a major medical school in the U.S., I am used to identifying and using statistical measures of such bias in scientific research.
In Japan, I have read that a product reviewer who writes for magazines or websites are paid fees by manufacturers. I have noted that a similar thing may be happening here in the U.S., both reading TAS, Stereophile, etcetera, as well as noticing comments from individuals on this and other websites, many of whom are also dealers of these products.
As an example, I am somewhat of a computer nerd and have been downloading high-resolution audio files for almost a decade. That being said, I have been looking to buy a relatively high-end SACD player for my large collection of CDs and SACDs. I have noted the following:
1. There are few-to-no reviews of DCS players (e.g., Puccini SACD player, somewhat outdated but can be upgraded) and almost no published U.S. reviews of the Marantz SA-10 SACD player that was released about a year ago. In contrast, SACD/CD players including those from Esoteric, Hegel (CD only), Ayre, PS Audio, MBL, and other brands commonly appear in formal reviews, which are all favorable. Does this mean that products which have been reviewed but which are not well-liked by reviewers are not published?;
2. Comments in this and other forums mention that one or another SACD player or other product "must not be that good because they appear often as used equipment for sale..." or something to that effect. This observation may be valid, but could easily be confounded by the number of such products that were, or are, available for sale. The greater the number of products, the greater the likelihood they will appear as used items for sale - it says nothing about the quality of the product. I like to call this the "Ferrari effect", as this manufacturer intentionally limits the number of cars of any model for sale, and the company often only sells to individuals of affluence and/or have purchased cars from them in the past, artificially inflating the value of these cars;
3. Odd statements about the interesting MQA file format, part of a larger problem of a lack of objectivity in the audiophile community. Recently I read in a publication - "MQA is to conventional audio what quantum mechanics was to classical mechanics" - Really? Does this individual know anything about physics? Or am I taking this all too seriously?
I guess I am asking about the degree of bias in these reviews, to what extent are products reviews influenced by the manufacturers and dealers, and where is the objectivity in this domain?
Thanks for listening to my ranting...Gerry
- ...
- 64 posts total
| Post removed |
Another reason why reviews are almost always favorable is because the magazine and or reviewer doesn’t really want to kill a small company, especially in view of the fact that sometimes things happen, you know, such as damage during shipment to the reviewer, operator error, failure to follow instructions, that sort of thing. Also, sometimes there are trade-offs for the device under review which would be spelled out in the review. For example, cost, size, complexity, peculiarities, etc. Finally, reviewers and magazines are not all the same. Reviewers and magazines can certainly have unique or different approaches to conducting reviews. It certainly wouldn’t be unusual to see products reviewed that are trending or that the reviewer and or magazine would suspect of having interest for their readers. |
I am not trying to insight a so-called flame war here, but I have to agree with willemj. If audiophile magazines are "hobbyist" journals, then I would stop the product reviewers from including what appear to be plots of measurements of impedance versus phase, spectral decay, and amplitude versus frequency. I enjoy looking at these graphs but in almost all cases, the reviewers will only hint at issues (e.g., speaker impedance spikes or drops), but the review will end with statements such as "best speakers I have heard, so I bought the review pair" and so forth. I can only believe that there is some economic incentive that is not disclosed. Comparisons to a hobby and a profession: 1) I used to be involved in Porsche club racing, so I would read porschephile and generic car magazines. This would certainly be considered a wealthy person's hobby, but the car magazines would provide head-to-head comparisons of performance between brands and would make disparaging statements about cars that were poor performers. I have never read head-to-head comparisons in a given category of hardware in any high-end audiophile magazine (e.g., different speaker brands in the same room at the same time, driven by the same audio hardware); 2) In biomedical research, there are many problems and challenges, but to suggest that our collective intent is to do anything but try to arrive at so-called "ground truth" is not accurate. That does not mean there is no publication bias (see [1]), but at least there is recognition that this issue exists, and many attempts to deal with the problem. - Gerry [1] A. César-Razquin et al. A call for systematic research on solute carriers. Cell. 162 (3), (2015), 478-487. https://www.snijderlab.org/pdf/26232220.pdf |
You missed the elephant in the room. Ordinary bits of wire. Not long ago, bits of wire were correctly viewed as bits of wire. Today gargantuan sums are paid for ordinary wires in the misguided belief that ordinary bits of wire do more than anything else in the audiophile’s setup. The fact is they don’t. What they do is lead to massive profits and mark ups for something cheap with no substantive benefit to the user. Audio reviewers and press have jumped on the band wagon (incentivized by advertising) and stores have jumped on the opportunity to sell additional cable trinkets and jewelry at extremely high mark ups to buyers. The advertising lie is that “everything affects the sound” and that “exceptional ears and exceptional systems will require these trinkets to sound their best”. A story just like the emperor’s new clothes which audiofools eat up readily. Smugly the audiofools look down on anyone with less than $1000 bits of wire and snear at those that don’t hear a difference as being deaf or not having a high end system. The whole domain is like high end skin care - packed with excruciatingly expensive products that do nothing more than appease the ego of the vain. The bits of wire are made of fairy dust meme also undermines real audio innovation. Are you at Johns Hopkins? My daughter is studying there currently. |
- 64 posts total