Audio Science Review = "The better the measurement, the better the sound" philosophy

The most respected audio designers listen more with their ears than a graph.  

Amir,The glass is more like the electricity which should be the neutral fundamental.A better analogy for the audio component is arguably the special roasted oak cask and the vintage process used to concentrate and enhance the base grape juice flavours.

But either perspective is fine depending on taste.Some people enjoy young unoaked wine and some don't.Most prefer something less natural but with a bit more sympathetic complexity.

I repeat....

Always follow the Money... right into pocket of Amir-asr 

Sorry, I do not Buy whatever it is that he is selling. Certainly it is Not, testing equipment for the altruistic benefit of those audio enthusiasts and wanna be engineers who can only see their way to spending just enough, to achieve Audio Nirvana .

I call BS...

Hey @amir_asr, I know there are a lot of tough questions here but no one is banning you for your answers. Please bring down the wall at ASR, let people that don't conform to your dogma (and don't pay for a name badge on their avatar) discuss respectfully. I think you will have more members if you include dissenters and not take a big hit to your cash flow, all good. This is not a one way street where you are right and that means everyone else must be wrong. 

What I find interesting is they won't acknowledge that measurements came about by simple observation first.

No one responded to my comments concerning use of the most exacting computer modeling and measurements to construct the $850 million orchestral hall that is very inferior to 19th and 20th century smaller, shoebox designed halls with zero computer technology and very basic mathematics.  Great posts today from Agoners.  

Well, I'm reminded of the old computer maxim, "Garbage in garbage out".  Just as true today apparently, as in 1957.

 

Let me first tell you that there is far more to conclusions we draw than measurements. Myself and many members of the forum are engineers and understand how your audio devices operate. We then combine this with careful measurements. And then look at what audio research (published) tells us. If all three arrows point the same way across testing multiple categories of products, then we have very high confidence in our conclusions as to efficacy of such claims.

As an example of above, we know how power supplies work in audio products. So when someone says this power cable "filters" noise that then does the same in your audio output, we can analyze this on all fronts. We know that there are multiple filters working far more effectively in your audio gear than anything a power cable (or conditioner) can do. We then combine that knowledge by showing that said power cable provided no filtering. And even the company itself showed no such evidence. We then go further and produce highly distorted AC waveform and show that the audio gear did its job and nothing changed in its output. After testing a number of such products with the same outcome, we then have a very high confidence answer with strong data to back it.

Please note that this is VERY different than what other objectivists do. I put in tons and tons of effort in testing these audiophile claims. I have tested more interconnects and power conditioners than I can keep track of. And when a new one is offered to me, I test it again in the thought that it may be the one that shows a difference. This should show you the openness I bring to this field. There is nothing "cult-like" about what we do.

Note that there situations where measurements provide part of the answer but not all. Speaker and headphone measurements are very powerful in their predictive power but not sufficient. We don’t for example fully know the effect of radiation patter for a speaker in different rooms and for different people. Measurements do however rule out the poor designs and do so with authority. Maybe some of those are still good but there are so many good choices with good engineering so why take a chance?

I recently recommended an IEM. A bunch of people purchased it. About 70% love it and can’t imagine how great this $50 IEM is. 10% to 15% say it sounds good but better with EQ. 10 to 15% say it is not for them. This shows how powerful imperfect or incomplete measurements can still be.

my question @amir_asr was whether "if it cannot be measured then it doesn’t exist" is an accurate description of your views.

all of the above is perfectly fine and i don’t take issue with any of it. and i appreciate you answering at length.

but i am genuinely curious whether you believe there are real-life auditory experiences with recorded sound that simply fall outside the purview of your approach to evaluating gear. in other words, do you think that everyone who hears a difference that isn’t reflected in your tests (to say nothing of the tests of other measurement-focused reviewers like goldensound and erin - guys whom i’m given to understand have been banned from your forum for mysterious reasons) is simply delusional? or is it possible that this phenomenon is just an obvious example of the incomplete/imperfect measurements you refer to in your last paragraph?

Pretty sure I can be a drive-by poster if I want to be.

Regarding electronics, there can be no debate. You put it on the bench. If the output matches the intended output. Then it is good. Else, it is bad. Example: amps turn little waves into big waves that can sufficiently drive the load as if they are an ideal voltage source - which all classAB or D amps that cost >$100 can do now. That’s it. It’s just too easy to screw up, and @amir_asr is one of the best in the world at not screwing it up.

Regarding analog/digital interconnects, there can be no debate. It’s just simple physics and/or EE theory. It doesn’t matter, but I’m an MSEE w/ 25 years experience designing analog and digital chips. I know the math better than most.

If you want to debate speakers with me, then first watch this video on Floyd Toole’s landmark research (link below). @amir_asr can correct me if I’m wrong, but he uses an implementation of this research to predict speaker performance, along with subjective listening to verify expectations. I haven’t read all the reviews, but I don’t recall anything he’s heard that wasn’t predicted. If Spinorama says it sounds good, then so will blind listening tests. Other than that, as @amir_asr points out, the only other thing that matters is if it can go as loud as you like it without hitting large-signal limits.

If you don’t believe in blind A/B/X testing to verify your assertions, then please don’t debate anything with me. You’re an audio theologist with money to spare, not a scientist.

@smprather you said, "there can be no debate", is this why you came to a forum that hosts debates about audio?

Let's start with a discussion then. You saw @tonywinga post earlier, describing his system, the effort he took to make it the way he likes, the pics he posted. You saw @fleschler post about his background and experience. Let's just start with telling us a bit about your system and what you want to accomplish, OK? Feel free to start a new thread if needed. 

@kota1 I believe that you're asking too much from Amir and ASR. Amir has a firm grip on ASR. Amir has strong convictions. Amir is intentionally deaf to opinions expressed which go against his convictions. And that's absolutely fine because Amir is master of the ASR forums. If you don't like ASR, join another forum. Or find another hobby. The world is your oyster, don't sabotage yourself trying to change Amir. 

Is the Hi-Fi hobby threatened because ASR is a one way street and Amir is the Lord of that street? Ridiculous. Find another street. 

Amir welcome. So are these negative posts having a negative impact on  your collection basket?

Thank you.  We don't have ads on ASR Forum. If we did, negative and bickering posts would help with that in how it increases page views.   So you have that backward on two fronts.  The fight here is helping the owner collection basket. Have you filed protests on that?

@kota1 I believe that you're asking too much from Amir and ASR. Amir has a firm grip on ASR. Amir has strong convictions. Amir is intentionally deaf to opinions expressed which go against his convictions.

 

I am not deaf to anything or I would not tested huge number of ineffective audiophile tweaks.  The more data the better.  Some of you rather not see this data. It angers you even though you lack technical knowledge of the field. You could choose to learn. But instead come here to complain, acting like you know more than whole of audio science.  So look to yours6fir character flaws.   That is where the answer is if you want to advance your knowledge of audio.

Amir, your detractors here have a collective motto: “Don’t confuse us with the facts because we know we are right.” It is thus improbable that they will give you a fair hearing.

The hobbyists and audiophiles on this forum comprise people that are both highly intelligent and highly motivated.  Many have a background in music or audio or both.  These are creative people that can think beyond linear A to B.  This is the type of creativity that drives innovation and change.  Discovery and ideas almost always come before understanding.  

Now we have a guy with no credentials in either music or audio walking around with an ohmmeter claiming these people are deluding themselves.  Of course that is a trigger point.  Galileo was imprisoned by the Catholic Church for saying that the earth was round.  The leaders of the day had to protect the public from this "deluded" man.  They refused to accept other possibilities.  Technology has changed the world but some people never change.

There was a story in the early 1990s about GM hiring a former car thief to try to break into their new model car featuring the latest in anti-theft technology.  GM engineers were so confident in their newest anti-theft deterrents that they put a large timer on stage and unrehearsed had this former car thief attempt to break in and steal the car in front of an audience of car dealers.  The story goes that the former thief walked on stage with a sledgehammer and hit the front bumper.  The airbag deployed and the doors immediately unlocked.  The guy got in the car with a screwdriver and started the car.  The large timer showed just a few seconds had elapsed.  Yes, the engineers had egg on their face.  Linear thinking has its advantages but also has limitations.  

@amir_asr, the walls are still up at ASR which is fine. If you came here to learn about audio that’s fine. If you came here to defend your reputation that’s fine. But if you came here to start an argument about why you are perfect and we are not let’s just set that aside for your forum at ASR.

The walls are still up there, you have really not discussed anything about YOUR system, what you are trying to achieve, etc. When you say:

"But instead come here to complain, acting like you know more than whole of audio science."

It comes off like

"... acting like you know more than me (being Amir_ASR)"

and you are referring to yourself as being the whole of audio science.

You still have not shared anything about YOUR system. Is it perfect? What do you own, why did you buy it, what are you trying to achieve.

There is no need to feel shy, we won’t talk down to you if it is built with components that are more entry level. You will find that many of the members here start with where ever they are now, take the collective "experience" (not just data) of the members here and increase their enjoyment of the music. Have you participated in any threads here besides this one? I think you’ll enjoy it.

It is obvious that Amir does not want to learn. This is a pity as his appreciation of music could really grow. There are many people here who are more qualified than he is. However never mind. Some people are just very stubborn and inflexible.

"But instead come here to complain, acting like you know more than whole of audio science". There are many people here who know far more than you. Also we do not come "here". YOU came "here".

Reminds me of the Commissioner of the United States Patent Office who in 1899 said the office should be closed soon because there was nothing left to invent.  

@laoman I find it interesting that Amir did not answer one of my questions. I wonder why?

Maybe Amir chooses to respond only to people with higher functioning cognitive abilities. Not interesting at all for those here who watch what you post.

Who claims there’s nothing left to invent? Immersive audio recording and playback is a relatively new invention and I imagine improvements will continue to be invented. DIRAC is getting reading to release a new invention of their room correction. A new type of DAC chip or amplifier could be invented but I don’t see much direction there as those available now are capable of reproducing the medium beyond human audibility, doesn’t mean someone somewhere isn’t working on something new. I believe Atmasphere a contributor here invented a new type of class D amp.

There are probably more than a few here who miss the good old days of arguing over Covid vaccines.

How can anything new be invented if we already have perfection according to the measurements you guys often talk about. 

@kokakolia , we will see how amir responds, I hope he didn't come here just to plug the leak in his collection basket. So far he has not posted anything about his own system, his tastes, what he wants to achieve. I don't think he has discussed anything audio related in any of the many other threads here. He certainly has not created a bridge between our forums and is still banning dissenters at ASR. Maybe he is a bit defensive still and will open up a bit and relax later and discuss more about his system, his tastes, etc.

How about some empirical data points.

Context: Long time audiophile, heard many multi $100K systems, my own system well over $100k.

Cocurrently owned Auralic Vega, LKS 004, Okto Dac 8 stereo, Musetec MH DA005. I know ASR reviewed both Okto and Musetec, Okto superior signal to noise and jitter measurements, Musetec did rather poorly compared to virtually every dac they've measured.

My experience with all these dacs, Musetec superior resolving, transparency, at least equal to Vega and Okto in ease, relaxed qualities, far superior dynamics, especially micro.

Based on ASR measurements, this dac should be relatively low resolving, transparency, have nervous, digital sound quality,, small sound stage. None of these things relate to how it sounds.

ASR signal to noise measurements of dacs has virtually no bearing on real world listening as analog section in audio systems higher noise. I see virtually nothing in their measurements that gives much insight to quality of dac power supply, things like authority, micro dynamics, or how it impacts flavor. More interested in jitter measurements, while 005 poor jitter measurements don't translate to sense of digititus or small sound stage, would be insightful to see improved jitter with this unit and see if that translated to even more ease and larger sound stage.

005 has been also compared to at least one other dac they've tested, vs Mola Mola Tambaqui, another superior measuring dac, 005 preferred in at least one case. 005 has also been compared to Weiss and Holo May amongst others, held it's own in all company. Based on ASR measurements this dac sucks, not true in real world.

It would be enlightening to see far more of these empirical data points, comparison reviews of dacs ASR has reviewed would give us far more insight into possible correlations. My take is this particular measurement protocol tells us very little about how these dacs sound in real world.

Take a look at this post on sound signature of amplifiers

https://www.audiosciencereview.com/forum/index.php?threads/measuring-the-sound-signature-of-two-different-integrated-amplifiers.37963/

ASR is like Audiogon.  There is a wide range of opinions on Audiogon the same way there is a wide range of opinions on ASR.  Amir pays for the website and offers his opinions.  Other readers like me can provide my opinions too, and it’s not deleted or censored and can be put up to the front page too.

What makes ASR different is that opinions should be backed by something that is not purely subjective nor purely relying on the trust of the an expert opinion.  Amir’s standard test set might not capture every last bit of potential for audio performance, but it is reliable and reproducible and FREE. Ground loops can be a real difference in home environments and there are times that I can get product A to perform just as well as product A in Amir’s lab, but product B in my home measures worse (and potentially sounds worse).  This is where testing at home is important.

30 years ago, we said that it was impossible to measure certain things we can hear.  The software and hardware has gotten better.  We can measure differences.

@kota1 I'm on the same page. If you were in the same city as me I'd share a drink with you. And Amir as well.

@gxalan, welcome to the forum, I see this is your first post. Can you tell us a little bit about your system, your taste, and what you would like to achieve in this hobby? If you were on a desert island and could only bring three CD's, what would they be? If you had a fire and could only take one component from your home what would you take?

@kokakolia , thanks for the reply, maybe we will meet at an audio event in the future (it could be virtual but the champagne will have to stay on ice.)

i find the "he’s just doing this for money" critique ridiculous. way to offer up a perfect mirror image of the bad faith argument you can see on ASR every single day. "we must conclude that by making poor (according to my arbitrary tests) design choices, these manufacturers are clearly only out to deceive gullible audiophiles" "that’s just snake oil" etc : "amir is only in this for money". it’s the same stupid argument from both camps. better to judge peoples opinions on their merits instead of reading malice into every little thing you disagree with

There are probably more than a few here who miss the good old days of arguing over Covid vaccines.

 

@kota1 I am a long time user of Audiogon to buy and sell gear and have a nice collection of vintage and contemporary gear.  The beauty of measurements is that you don’t need to rely on taste or my gear list. :)

I enjoy a lot of different types of music and enjoy different systems depending on my mood.  I have owned Proceed/Levinson gear, contemporary and vintage McIntosh solid state and tubes, Pure Class A Accuphase, PS Audio, Nelson Pass era Adcom, Primare, NAD, Sony ES, Denon, Marantz, etc.  I have had planars, ribbons, omnipolars, soft dome tweeters, metal tweeters, compression drivers, etc.

I find that the most accurate sounding systems are the best for well recorded classical music but are too unforgiving for good but not great singers or good but not great recordings.  (Examples of a good but not great singers are Emma Stone and Ryan Gosling from the La La Land Soundtrack, and a great singer with a poor recording, Joyce Jonathan’s Sur Mes Gardes.)

As a music lover, I have achieved everything I need for great sound and enough experience to pick the system that gives me the most enjoyment for any individual piece of music.  As a hobbyist, and why I spend time on ASR, is that I hope to show everyone that everything that can be heard can be measured, it’s OK to prefer coloration, and there are a lot of tweaks that are just sighted bias/snake oil.

Anyone who is serious about the hobby of wanting to know why something sounds good (or bad) should invest in their own test gear.  Compared to what some are charging for cables or line conditioners, it’s pretty cheap. 

Well, ASR members are going wild over at ASR on their rebuttal forum with 165 posts.  One member lists this statement to prove his point: "The inability of some posters there to accept evidence-based science is stunning. It is a similar logical construct as saying “I’m not going to treat my cancer because the treatments make me feel worse. Your fancy MRIs, blood analyzers, and pathology tests can’t measure my feelings.”

"Then he lists uses of technology: If they DID say that, then they would at least behave in a consistent manner. But they DON'T behave in a consistent manner.
Realistically speaking, they will use the science-based medical technology. Ergo, they trust it.
They will trust the science-based technology that goes into their cars, hybrid or otherwise.
They will trust science-based aerodynamic technology when they travel by airplane.
They will trust the science-based digital technology that enables their refrigerators and stoves to operate correctly.
They will trust the digital technology that allows their TV to entertain them.
They will trust the technology-based science that warns them of hurricanes, tornadoes and storms that are broadcast on said TV.
They trust the science-based digital technology that allows them to use their cell phones.
Most of them even trust the science-based use and control of nuclear power, so they can enjoy the benefits of electricity."

His insane conclusion about audiophiles not subscribing to ASR dictum:
"They just reject evidence-based science when it come to audio.
Selective little buggers, aren't they? " 

Of course we all know (except ASR members/Amir) that we do not reject technology and science.   Again, analogies that are replete with incompatible comparisons of sound equipment with wine glasses, etc. that posters make and Amir indicates the fallacy of their thinking.  They cannot separate the innately technological for products that are consistent in form and factor from the ephemeral, like wine.   Sound reproduction is NOT finite.   The equipment obviously (to us) has sonic parameters which are not being tested.  If it were, then all equipment that measures the same would sound the same.  Hence, there are a great multitude of equipment choices as well as synergy aspects to consider which are also, not tested.  Imagine if there were a million choices in cancer treatments?  How about the long list of trusted technology information above?   No, there is a clear cut difference between sound reproduction and science only based information and products. 

@gxalan , thanks for the reply. I have a thread going elsewhere on this forum and I need advice about a mcintosh streamer/dac vs a just dac, feel free to post something over there, appreciate it. I have never tried mcintosh and am thinking hard about it.

@rtorchia @tonywinga @kota1 @kokakolia @td_dayton @laoman @djones51 and others on this forum. We are a more intellectual group, willing to converse in terms of possibilities in reproduction sound equipment that can be different yet sonically appeal to each of us, even in different ways. Measurements can find faults and limitations (as well as shoddy construction/materials) in equipment, they cannot determine how they will sound compared to one another or in an AUDIO SYSTEM and IN A ROOM (both of which vary more than the possibilities in chess.

Amir appears to be sincere and I don’t doubt that he is not exclusively in it for the money. That does not equate with the nonsensical diminution of non-ASR audiophiles who listen to different music, differently in different systems in different rooms.

Then there are personal choices that run counter to the major audio magazines. I used a Benz Ruby 3 for over a decade with an audio interface SUT with matching resistor impedance. I purchased a Zesto Allesso SUT which sounded slightly better using trans matching impedance. When I replaced the worn cartridge, I sought something that would play most of the 28,500 LPs and not just be outstanding on 20% of my collection. My friends were gaga over the Dynavector 20x2 H which I also heard great sound from. I purchased the L unit since I had the Zesto. At max gain and 100 ohms, it was a perfect match, including my modified SME IV arm. It doesn’t have the very wide separation of a $4K to $18K cartridge (many do) but the tonal balance is to my liking despite missing some resolution.

Here is the choice, resolution over tonal balance. I would not own a Lyra as I consider them tipped up in the highs (compare the measurements). My Dynavector has a flat frequency response. I like that. See-measurements can help determine something. But that was only a starting point. Without my Zesto SUT, I may not like the Dynavector low output as much in my system.

@fleschler +1

ASR love to quote Floyd Toole and research done at Harman.Which is fine but the findings there have also shown that the average listener does not prefer a flat frequency response .Far from it.And different types of listeners prefer different frequency response curves and these are only averages anyway so do not properly reveal the extent of this variation.So there is substantial variation between different people and how they hear or listen and what they prefer and yet the hardcore objectivist tends to obsess about electronic components needing to have ruler flat frequency responses and also makes the assumption that everybody has a room that can reproduce a flat frequency response.And yet if you dare suggest that not everybody lives in an anechoic chamber you will be banned from commenting.It as if they want to construct and inhabit some sort of alternative reality based on theoretical parameters and totally ignore real world variables and personal differences.

I do wonder if a lot of them have spent too much time behind a computer screen absorbed in online alternative reality and are not comfortable with the foibles,complexity and diversity of the real world and real people.For them a theoretical construct is going to  be where they feel comfortable.

You’re an audio theologist with money to spare, not a scientist.

Thank you, yes I am. A scientist knows that a tomato is a fruit. The food artist knows not to put a tomato in a fruit salad. 

There is an art to putting together a great system. The person guided only by measurements will often create an unlistenable system with each piece of gear having outstanding measurements. At least that has been my experience.  

@amir_asr

Welcome to AG, and thanks for responding to some of the questions raised here.

Alas, I suspect that, despite your best intentions, that earlier post by @rtorchia will prove prescient.

"Amir, your detractors here have a collective motto: “Don’t confuse us with the facts because we know we are right.” It is thus improbable that they will give you a fair hearing."

@smprather

"If you want to debate speakers with me, then first watch this video on Floyd Toole’s landmark research (link below).

.....

"If you don’t believe in blind A/B/X testing to verify your assertions, then please don’t debate anything with me. You’re an audio theologist with money to spare, not a scientist."

Well said! Your request seems sensible enough for anyone with a genuine desire to educate themselves with at least some of the fundamentals of loudspeaker science, but again, alas I don’t see many takers so far.

Perhaps others here could chime in so we can establish a broader consensus?

@jtgofish @russ69 @cd318   YES! 

When I play a chorale piece on my stereo system the choir fills the front half of the room floor to ceiling. I can hear individual voices across the soundstage. It is not the ultimate in resolution- I have heard systems that can resolve each and every instrument in an orchestra spread out across the room. As I work on my system with both upgrades in gear and with tweaks, I find that the resolving power of the system improves. That chorale piece has evolved from a cloud of sound years ago to individual voices. Its amazing to hear. I have no idea how one would measure or quantify that property of the stereo objectively. I found not just upgrading preamps, amps, and cables improved resolution but isolation and room treatments as well reduce the noise and muddiness or blurring of sounds. Now I can measure the mechanical isolation of my stereo components with an accelerometer and FFT analyzer and I can see that they are isolated from vibrations above 3 Hz. This results in improved resolution and more detail in the music. In addition to those upgrades I also find tweaks to the digital streaming ethernet improves both sound and resolution. It would seem that could be measured but no one seems to be able to get past, "it is just 1’s and 0’s". There seems to be some 2’s and 3’s getting through. :)

It seems naive to just look only at distortion, frequency response or time domain properties. Live music outdoors is like a hemianechoic chamber, ie. no reflections. So the highs roll off very quickly. The same for a concert hall. Someone who likes that kind of music is not interested in strong bright highs. On the other hand, someone who likes to listen to live Jazz or Rock with electric instruments wants to hear the highs. Plus, the same speaker in a different room will sound different. Leave those types of measurements to the manufacturers. The boutique manufacturers voice their gear to satisfy the bulk of their clientele. Not every brand nor every model will be to everyone’s taste.

I have couple questions to “ASR Tests Originator”:

1: Can you please describe your measurement equipment' employed at ASR LAB? I assume tester is calibrated, maintained, and upgraded periodically, - correct?

- are you doing your measurements in Faraday Cage?

-what cabling, loading, power, additional filters, shielding, etc are used for ASR tests? ..and why do you think your test setup is good to represent average user?

2: How do you select units for tests? Do you receive devices from manufacturers, or do you obtain those randomly?

- sometimes manufacturers prefer to send “good units” to third party test LAB to produce “excellent reviews”, while most of manufactured units of the same model are "so-so". I haven’t seen two analog sound units, including speakers, amplifiers, pre/eq/etc of the same model/brand with the same test results in my life!

regards

OK,how about...

I Am Science...

Where have I heard that before...?

@decooney

The most respected audio designers listen more with their ears than a graph.  

How on earth would you know the list of audio designers and the "respected" subset?  And how do you know that is a truth as opposed to a talking point post poor measurements of their gear?

I can also say the opposite. How would anyone know if you are right or I am?
 

I will answer: you need to understand electronic design. If you did, you would know the above claim can't possibly be true. Modern electronic design starts with simulation and full analysis of circuit performance including things like distortion, frequency response, etc. Then prototyping starts with the designer's eyes glued to instrumentation like voltmeters, oscilloscopes, and if we are lucky, audio measurement gear. If they lack the latter, heaven help us as they have no idea if they are designing something performant!

But let's say you are right. So what? I am supposed to trust the ears of a Joe designer?  They could have all the design expertise in the world. It doesn't mean they have critical listening skills and know how to compare audio gear sound without bias in controlled listening tests.

Bottom line: you are falling for marketing lines. Demand proof that their equipment is well engineered and transparent to the source. Anyone can say anything.  Ask for reliable, third-party verification. Don't worship heroes.  Insisting on validation.

Amir-again dumb retort.  Of course good designers use test equipment to design and test their results.  However, if it ended there, they are making HUGE mistakes without knowing it.  Speaker designers who don't listen to adjust settings are not very good designers.  My example of measurement uber alles for making the most expensive and as good as it should be at $850 million orchestral hall just confirms that there is more than measurements in design.  After most modern orchestral halls are built (including that one and my local Disney Hall), they nearly always undergo significant renovation to make them sound better.  Why, because we hear the results and adjust afterwards which can include new measurements and cannot rely on only scientific results. 

@westcoastaudiophile

I have couple questions to “ASR Tests Originator”:

1: Can you please describe your measurement equipment' employed at ASR LAB? I assume tester is calibrated, maintained, and upgraded periodically, - correct?

You assume wrong. We are not interested in metrology, or measuring things with repeatability to 5 decimal places. Typical SINAD (noise+distortion vs signal) varies by a 1/10th of a dB as the gear is sitting there.

That aside, call Audio Precision and ask if their analyzers require calibration. They will tell you that only if  you are doing government work and such and need such "cover your behind" certificates.  Measurements I perform are routinely replicated by manufacturers and other third-parties.  

That aside, I have a long list of equipment in the lab. Key ones are:

Audio Precision APx555. This is a $28,000 state-of-the-art analyzer. It is used for testing DACs, headphone amps, amplifiers, pre-amps, phono stages, etc. I have been a customer of Audio Precision since early 1990s when I worked at Sony.  AP is the gold standard in audio measurements.

Headphone measurements are based on APx555 above with addition of GRAS 45-CA measurement rig ($14,000).

Speaker testing is performed using Klippel Near-Field Scanner from German company. It is a robotic system for measuring anechoic response of a speaker in 3-D space. It costs roughly $100,000.

These are the core instruments. Beyond them, I have a ton of other gear from scopes to meters and everything in between for specialized testing.

I would say if you had to put together a lab like mine, you would need about $200,000 of capital investment plus tons of specialized knowledge and experience.

- are you doing your measurements in Faraday Cage?

-what cabling, loading, power, additional filters, shielding, etc are used for ASR tests? ..and why do you think your test setup is good to represent average user?

2: How do you select units for tests? Do you receive devices from manufacturers, or do you obtain those randomly?

No Faraday cage. Those are used for EMC testing.  None of you  use your stereo gear inside such a cage so I don't see why I should test them that way.

Cabling for XLR is Mogami Gold. For RCA, I use Amazon Basics. No power conditioning is used or needed. I have tested a ton of these and either do nothing, or impair performance. I do have a LAB AC generator that I use for special testing.  Ditto for DC lab generator.

Vast majority of audio gear that I test comes from members or those that I personally buy. Manufacturer gear is 5 to 10% of the total. I routinely test duplicates sent by members and have yet to see a case where the company sent unit is golden sample. A number of companies also offer that test units be purchased randomly if there is lack of trust.

My oscilloscope doesn't listen to music.

@fleschler

Amir-again dumb retort. Of course good designers use test equipment to design and test their results. However, if it ended there, they are making HUGE mistakes without knowing it.

Well, the dumbest thing is to take the word of a designer attempting gorilla marketing in forums and such. Why on earth would you trust what he has to say about his own gear? Isn’t he totally biased to get you to buy his gear? Would you trust a drug from a drug company with no government certification and controlled testing? Why would you put your guard down when it comes to audio? What happened to common sense?

And no, they would not make "HUGE" mistake if they stopped at instrumentation. As I explained, there is no reason to know they have good ears whatsoever. On what basis you put any value on what they think of their own gear vs another? He knows how to design electronics but all of a sudden, he has golden ears too?

Nothing wrong with claims about fidelity. But that better be in controlled listening tests published with full documentation. Him saying he tried many things and the one being sold is the best is hogwash.

I tested a PS audio directstream DAC. It measured poorly in low frequencies. After a bunch of back and forth with claims you believe in, it turns out that for cost reasons they used low quality transformers that produced more distortion! Multi-thousand dollar DAC used low quality parts. So you better not believe company claims. Your trust should only be on what you can verify. And I am here for verification. If you don’t like that, setup your own controlled listening tests. These are the only proof point that respected audio journals accept. Don’t make up your own methods. They only serve to confuse you into spending money after bad.

@invalid

My oscilloscope doesn't listen to music.

Then you don't know how to use it or it is broken.

@fleschler 

 Speaker designers who don't listen to adjust settings are not very good designers.

I am sure the person I was responding to was talking about electronic design. Of course speaker designers listen to their speakers. Key is whether they do that using controlled testing or not. And whether they use proper set of measurements in addition to that.  If the answer is that they don't use such measurements and only use their own ears to figure out what is or is not good sound, it would be a speaker I would stay away from.