As good an analogy as any, I guess.
I wrote off ASR long ago.....
The Audio Science Review (ASR) approach to reviewing wines.
Imagine doing a wine review as follows - samples of wines are assessed by a reviewer who measures multiple variables including light transmission, specific gravity, residual sugar, salinity, boiling point etc. These tests are repeated while playing test tones through the samples at different frequencies.
The results are compiled and the winner selected based on those measurements and the reviewer concludes that the other wines can't possibly be as good based on their measured results.
At no point does the reviewer assess the bouquet of the wine nor taste it. He relies on the science of measured results and not the decidedly unscientific subjective experience of smell and taste.
That is the ASR approach to audio - drinking Kool Aid, not wine.
46 responses Add your response
Another ASR hating thread. One need not say that listening does not matter in order to see the value in measurements. Are there ASR folks who think that only measurements matter? Sure. But that does not mean that the measurements done cannot be helpful at all. (I mean, I’m not a fanatic about my weight on a scale, but it helps to know when I’ve gained 10 pounds.) I honestly don’t understand why ASR is like Voldemort to some folks here. |
Analogies of audio to other hobbies are generally suspect (especially the car ones), but yes I can sort of get on board with this one. The folks at ASR have a very different hobby than what is generally practiced here - though we’re superficially under the same title "pursuit high-quality home audio". Unfortunately that’s about all the discussion that can be had about the 2 camps before it goes way off rails. I remember reading an ASR "review" that was chock full of the useless SINAD analysis, and then for the "subjective listening" portion something like this was written: "we had comany arrive at this time, so no listening tests were performed". That's their hobby, in a nutshell lol. |
It would be really cool if somebody who wanted to criticize ASR could do it without strawmen.
And this is the case with loudspeakers. It’s like so many audiophiles imagined that measurements are just plucked out of the air for no reason at all. The whole point of measurements is that they have been correlated to how things sound. That’s the point of measurements! And scientific study has shown that certain measurements correlate to what most people will rate as higher quality. No it doesn’t mean you have to write the sound quality as high. But then it’s the case with every single review ever written, subjective or otherwise, that you might not like the speaker yourself. But it’s completely rational for somebody to use measurements that have been correlated to good sound as a way to winnow out the loudspeakers that they are going to spend time pursuing. |
+1 @mulveling |
There are any number or audio equipment reviewers that could be ignored just as easily as ASR. Then again any audio equipment manufacturer claims and advertisement could also be ignored as well as dealers hype. You provide a good analogy. I think it's always a good practice to never take any review/claim as " the truth set in stone". Do your own research and try to listen to a component before making an opinion. |
“Among a myriad of other complaints, I simply don’t have any confidence that ASR as a publication knows good sound.” ASR evaluates equipment based on objective criteria that has been found to predict certain aspects of sound quality. The speakers forces are rated based on the type of measurements that have been shown Through rigourous scientific testing studies To be correlated With the type of sound most listeners - That includes audiophiles and reviewers! - Will rate most highly.
This may not be perfect, But I’m guessing they are on much stronger ground Than you might be.
What Do you claim to be “Good sound “ And why should someone agree with you?
|
@prof |
This post was inspired by answering a question about power conditioners. In reviewing the Puritan Audio PSM 156 on ASR, the author says "as always, we attempt to tease out the transfer function of the conditioner using normal audio level signals" and after many measurements with test signals concludes with "as you see, I have run a number of tests to give the PSM156 ample opportunity to show it can do something to improve audio but it can’t even move the dial one hair. There is no indication or logic that would tell us that it can make an audible improvement." It is important to note that at no point was it mentioned that music was played through a system with and without the PSM 156 in the circuit. I don’t spend my hours of relaxation listening to test signals - I listen to music. The ASR review failed to assess the impact on music reproduction. |
I also think there’s a little bit of Kool-Aid at play within ASR. There was a poll conducted in Reddit’s r/audiophile which asked the following: You just purchased a MHDT Pagoda DAC that Amir from ASR said: “It goes without saying that I can’t recommend the MHDT Labs Pagoda stereo DAC whatsoever.” It however sounds lovely in your system and more to your listening preferences. You’re truly smitten by this poor performing DAC. What do you do? 1. Sell the DAC 2. Keep the DAC 62 subscribers to this subreddit would have kept this DAC while 20 would have sold the DAC. Imagine selling a DAC that sounds great in your system simply because a guy, who if I remember correctly, measured this DAC, never listened in his review, and subsequently couldn’t recommend on his forum.
|
I make no claims that suggest anyone should agree with me. I’m a vocal proponent of learning to choose on your own to suit your tastes. Fortunately, I don’t make a living suggesting audio products to others. I simply express my opinions on a forum when inspired to do so. When it comes to audio, I aim only to please myself, and I’ve come to trust my ears to do that, because that’s been my method for deciphering sounds since birth. It’s an amazingly sensitive natural system that can pick up on nuances...much tougher for microphones, measuring devices, and machines to pick up on those subtle cues that separate a sax from a clarinet. "Good sound" to me is what tricks my brain into giving me an aural experience that’s similar to attending a good live musical performance. There’s an emotional connection that inspires and captures my attention. It’s natural, organic, textured, full of nuance, motion and emotion.... unlike microphones, measuring devices, and machines. Even though it’s a highly personal selection that truly should be done by each of us, there are probably several members here who I’d trust to pick a system for me if it was necessary. Amir, would not be among them, because I simply do not trust that he could choose a system that would be satisfactory to me. Specs alone aren’t going suffice for picking my gear. |
+1 @prof Well said, well explained, well justified. |
+1 @2psyop Not sure why these ASR sucks threads keep coming up when half of the posters say they don't read ASR anyway. It's just information, and no more or less significant than written reviews without context or full of personal bias. Do your research, and then buy what you like. Do you buy wine based solely on a review? Reviews can spark interest, but at some point you need to open a bottle and drink it yourself.
|
As an electrical engineer who's spent close to 30 years designing and building hi-fi electronics and loudspeakers, I have quite a few things to say on this topic... I'll try and be concise. Objective measurements are great tools insofar as the results are understood and interpreted properly. This is where the current debate seems to run into trouble. Take SINAD (aka THD+N) for example. There seems to be a monomaniacal over-emphasis on this metric as an end-all-be-all measurement which somehow dominates the subjective performance of a piece of equipment over most other aspects of performance. Since we're doing analogies: SINAD is like a ruler. It measures one, and only one, parameter of the DUT. Just as a ruler can measure length but can do nothing to measure density, hardness, color, IR emissivity, chemical reactivity, unladen airspeed, or any of a thousand other possibly relevant properties of an object, SINAD / THD+N tells you nothing about most aspects of frequency domain or time domain performance of the equipment, both of which are extremely relevant to how it will subjectively perform. I would even argue that THD in the context of electronics is increasingly irrelevant, given how low distortion is in most modern designs. Turn to another famous objectivist like Ethan Winer, and you'll find excellent demonstrations of the audibility of THD. His demos area easy to find, and the shocking conclusion is that anything below ~1% THD is (or can be) essentially inaudible. Indeed the conclusion of the pioneering scientists in this field (close to a century ago) was that 0.1% THD represented a reliable threshold of subjective perception. And your loudspeakers are unlikely to do better than about 1% THD as well. So why chase after 0.0002% THD in a DAC or amplifier? I've built, lived with and loved tube amplifiers with rather embarrassing distortion figures compared to the modern benchmark. I've also built solid state amps following the guidance of famous objectivist Douglas Self, and while the result measured extremely well (~0.004% THD), the subjective listening experience of my earliest efforts was... disappointing to put it mildly. I kept using my tube amps while spending years trying to decipher how to build a better sounding solid state amp. So measurements have their sensible limits as well. It does no good to go overboard with a single specification. "But", you may quibble, "Amir at ASR has a whole suite of measurements intended to cover every aspect of good performance". While I commend Amir for his attempt to cover all the angles, IMHO his measurement selection still misses a number of relevant subtleties. Most notably the time domain performance of DACs and amplifiers, which are essentially not measured at all by instrumentation like the AP analyzers (they rely upon steady-state sine stimuli only). First-rate engineers like Bruno Putzeys and Julian Dunn have pointed out the importance of subtle things such as passband ripple in digital oversampling filters, and of reaching full stop band attenuation below the Nyquist frequency, yet these important aspects of DAC performance continue to be widely overlooked. Possibly because they are harder to measure and interpret the subjective effects of deficiencies? I don't know. But I do know this: a ruler (even a really really awesome one) is just not good enough. The human perception of sound is not well understood and even less well quantified, and there are many aspects of objective technical performance of audio equipment that we already know something about which are being overlooked. At the end of the day, it all boils down to this for me: Objective audio measurements must by definition be subservient to the Subjective outcome. If not, then we aren't talking about hi-fi anymore. As Nelson Pass so eloquently puts it: "We are in the business of entertainment, not dialysis." I am fairly sure that "the science" (or really: the science as practiced) will eventually catch up as better measurement techniques are created and gain widespread acceptance. It's just that at the moment, the audio industry as a whole has stagnated on a small set of decades-old methods. Part of this is a marketing problem too. The "objectivist" types are rightly critical of an unfortunate degree of snake-oil peddling and pseudo-scientific misdirection that goes on within marketing departments, and elsewhere under the guise of "subjectivism". Toss in a dose of Dunning-Kruger effect, and the usually deep innate desire of every audiophile and their ego to be an expert, and we have a different problem category, which the ASR types are doing their best to shut down. I applaud this aspect of their endeavor, but sometimes they take themselves too seriously. So take it from a die hard engineer and objectivist-subjectivist fence sitter: Trust your ears, and buy the gear that makes you happy. |
OK, that was long. So much for concise. Let's try again: Measuring audio gear and interpreting correctly what it all means without overlooking something important is HARD. After 30 years of ongoing study and practice as an engineer, I'm still learning new things all the time. The ASR crowd is well meaning and I applaud Amir's attempts to help educate the public - he is doing a great service, but lots of people get carried away with accepting his test results as the be-all-end-all last word. "It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so." –– Mark Twain Meanwhile, the high-end audio industry is unfortunately plagued by hucksters selling pseudo-science and snake oil. It's gradually improving in my opinion, but still a problem that contaminates the credibility of subjective reviewers. Ultimately, I believe the situation regarding measurements and their correlation to subjective results still has some distance to go, but will continue to improve. But until we arrive at the magical land where the numbers can tell you everything, just trust your own ears and enjoy the hobby. |
I have had the good fortune to work professionally with wineries and know a few wine/grape experts, including one whose family has a California winery. I have visited many and been on behind the scenes tours with wine makers and other professionals. What you describe is EXACTLY how wines are grown and made. The poetry about the nose and the personality and the terroir enter in the tasting and marketing phase. There is art but there is a very firm foundation of science. That is one of the reasons cheap wines these days are much, much better than they used to be. I think your analogy is a good one. The serious hifi designers and builders I know of design, build and test with measurements. They listen too. The poetry here also comes with the marketing and promotion part of the business. The one segment of hifi I wonder about is cables. It is possible that is all done with subjective judgement. ASR is a good reference and a useful service. Stereophile has the best approach to reviewing in my opinion. I like the poetry but pure subjectivity without at least an attempt to tie it to verifiable fact is useless to me. A good friend is an audio junky and also in the retail wine biz. I can attest selling wine is a LOT like selling hifi. There are always customers who think the higher price wine is always the best, and they won't settle for anything less :-) |
Colour, specific gravity, (residual sugar, salinity), boiling point of wine is like physical aspects of equipment in audio industry Eg size of drivers, cabinet crossover...(specification) Instrumental Analysis is of audio equipment ASR is doing in wine analogy using chromatography to list % of ingredients in the wine viz flavinoids, ethy alchohol and byproduct alcohols. Upon this data one can derive how particular wine can taste like. Audio industry need both Analytical and critical listening. Not no mention People by Audio equipment only after long listening at their Home.not based on review of any kind.
|
I really don't understand so much bashing of ASR. To the best of my knowledge, ASR is unbiased (it does not accept advertising) and provides accurate measurements. Measurements provide information -- clearly not all the information one might desire, but information that is potentially useful. Some equipment receives a positive recommendation; other equipment receives a negative recommendation. Compare this with Stereophile and The Absolute Sound, which are driven by advertising; indeed, it is common for the review of a piece of equipment to appear in the same issue as an advertisement for that equipment. And I have yet to see a negative review of any equipment. |
Why are so many "audiophiles" so obsessed with that site? Measurements make a difference but shouldn't be the only thing one uses to determine what to buy. I'd rather go by measurements than all the subjectivity that many (most) so-called audiophiles use to determine what sounds good and what doesn't. |
I'm not an ASR disciple, I've invested in some cables and don't believe measurements tell the whole story, but measurements at a minimum seem to add useful context. I don't buy into Amir's philosophy by any means, but his posts here have been civil, at least the ones I've read. Certainly more civil than many of the responses he gets. Amir, Gene and the other "objectivists" add useful information, but not the last word as their more pedantic followers assert. The ones I don't respect are the reviewers who say that measurements don't matter and blind tests are invalid. One suspects that the most important characteristic of a product for them is that they received a free copy to review. Lastly, I don't know about the wine analogy-I'm in the middle of bourbon country and that process is filled with chemistry and measurements. The difference is that if you have a set of cheap, poorly made speakers, the third song you hear through them will sound as bad as the first. On the other hand, as you get past the third glass of Kentucky Gentleman, it will taste more and more like Pappy. Burn in, I suppose.
|
I no longer have the listening acuity that I once had, so my days of having firm opinions about one thing sounding “better” than another are behind me…like a wine snob with COVID whose sense of smell has gone away. But as a person retired from a 50+ year HiFi career, I have opinions, some based on what I used to be sure I heard. I know that some audio designers believe as Amir does that audio component accuracy can be reduced to transfer function and SINAD. What else is there, after all (they wonder)? But I used to hear “depth” and “spaciousness” in some things, and not others, where both easily met criteria for flat, extended bandwidth and THD and noise. I even had a preamp that sounded positively marvelous…unless you engaged the rumble filter, which collapsed the soundstage. The engineer who created that preamp may not have noticed, or even listened to it to find out. If .1%THD in an amp or preamp is “inaudible”, why is .0001 better? The ASR “less is better” approach doesn’t impress me. He does base his recommendations on an array of measures and does recommend many items that aren’t “the best”, but there is a Calvinistic aversion to luxury build quality and pricing (if not matched with top class measurements) that is appealing to the less affluent, perhaps younger, reader. I used to belong to the Boston Audio Society, and these debates between objectivists and subjectivists were just as entertaining then. |
"To the best of my knowledge, ASR is unbiased (it does not accept advertising)." |
"Another ASR hating thread. One need not say that listening does not matter in order to see the value in measurements. Are there ASR folks who think that only measurements matter? Sure. But that does not mean that the measurements done cannot be helpful at all. (I mean, I’m not a fanatic about my weight on a scale, but it helps to know when I’ve gained 10 pounds.) I honestly don’t understand why ASR is like Voldemort to some folks here."
What you said immediately clicked the bulb in me. IMO, the error of ASR resonates a lot with the error of your analogy. About gaining 10 lbs. You could be gaining 10 lbs in water weight, Or 10 lbs in muscle. Or 10 lbs in fat, or any combination of these things. You can look quite different depending on how you gained your weight. You could weigh 210 lbs and be visibly/noticeably thinner than when you were 200. How is this possible? Well you could’ve gained extra muscle mass, lost fat in the process, and because muscle weights A LOT more than fat, you will look slimmer even though you are heavier. Kudos to you for making a perfect analogy to weight gain on a scale vs what ASR does. |
@prof Your argument relies solely on this simple equation Good measurements = good sound. Great measurements = great sound. Bad measurement = Bad sound. Different wording but it all implies the same thing. That if something measures good, it must sound good. Which is completely untrue and many can attest to this, Both consumers and designers. I have seen respectable designers come out and say they purposely use a worse measurement component because it sounded better than the better measured one. If measurement was a good indication on the sound quality. ASR should only measure products and not do any listening. The Topping D90SE was (is?) the best measured DAC ever, and how does it sound? Honestly not very good. Going back to the measurements. There are products that measure just god awful and they sound great. It doesn't take much to realize there's something amiss in the way ASR do their measurements. |
Thanks for sharing your invaluable knowledge. A quick word and disagreement on Amir. I don't think Amir means well. I've seen him spoken out so many times. His demeaner is "my way is the right way". Never a good trait for any scientist. What you said about SINAD reminds me of the TV industry these days. It's all about the nits and how bright the TV can get. Completely forgotten the many other important aspects that make a TV good.
|
I think a better analogy would be to measure the bottle and it’s ability to deliver a “neutral” end product. One that doesn’t change the flavor of the contents. Though in the audio context, we would have to agree on what is a “neutral” vessel/ measurement. And we don’t know what the “original content” sounds like to begin with. Measurements are still a good baseline though. |
I’ve always been fascinated about this part of amplifier design and the few remaining OG designers who still choose to use their ears to decide what sounds right to them for the final version that goes to production. As for the amplifiers that I’ve owned or borrowed and enjoyed the most, none really offered what anyone would refer to as spectacular measurements.
|
This post was inspired by a question posed by @kjl1065 in a post in Tech Talk titled 'Seeking a Power Conditioner'. In it the OP wrote: "Read reviews on both Niagra 1200 & Puritan Audio PSM 156 power conditioners and the reviews were extremely positive. While reading I came accross a review of both products by Audio Science Review (ASR) who claims his reviews are objective with scientific data supplied and his reviews were not nearly as positive to say the least. Anybody have any thoughts on how I should proceed with the differing of opinions." There was clearly an opportunity to voice an opinion about the ASR approach to audio reviews that in this case was based only on measurements but without any listening, and so the wine analogy was born and I thought it would be fun to expand on it here. This was not meant just an exercise in ASR bashing, but as an exercise in educating those who might not be as familiar with ASR that they should be wary of their absolute opinions. I believe that objective measurements are important, but so it the subjective listening experience. Objective and subjective balance and complement each other as we see in the Stereophile approach to in-depth reviews. |
Interesting discussion. I read ASR for a while because I found the reviews with data helpful. I also read and listened to reviews on other sites that discussed listening impressions. Put them all together and you get a sense if a piece of equipment is worth checking out. The thing that turned me off about ASR were some of the comments that were posted. If someone posted they liked a piece of equipment I would often see posts using measurements to challenge the post. Then I would see it get personal with comments back and forth that were very negative. I listen to music and read about equipment as a hobby to get away from the day to day grind and all the negativity that seems to be at an all time high. Life is too short. I don’t have time for negativity. One of the things I like about this site is I see that people share their opinions and they’re respectful and usually positive. As Hans says, enjoy the music. |
@analog_aficionado Outstanding post (the long one). Helps very much to understand and reconcile the differences between ASR results & listening. Thanks! I am in the camp that says ASR is a single source of information just like every other review. |
measurements are critical to audio, but a 100 grams of german chocolate measures the same 100g as 100g of dog crap. they dont taste the same. |
