Audio Science Review = "The better the measurement, the better the sound" philosophy

@prof I was only pointing out a prior Audiogon contributor who I found interesting who has been an avid ASR contributor but now is ridiculed and deemed unworthy of remaining because he said something different

I find that characterization quite misleading.  It isn't just because "he said something different" it's that he's making dubious yet-very-confident claims, and providing poor arguments for those claims!

OldHvyMec is frankly being treated mostly with kid-gloves in that thread!  Almost all (or all) the replies are quite civil and are simply pointing out the flaws in his argument.  One person asked OldHvyMec if he's sure he'll be happy at the ASR forum, but I didn't see a single person saying he was "unworthy of remaining."

(which most manufacturers of quality audio equipment believe is true, and I believe true of inexpensive equipment as well). Tell me a speaker or cartridge, very mechanical devices, don’t break-in.

The question is always "are the changes audible?"   There are good arguments, it seems, for why even speaker break in is over-hyped (e.g. most of the breaking in of driver surrounds etc typically occur rapidly, not over great lengths of time, though there seem to be *some* data suggesting *some* drivers can take longer to break in.  But this hardly supports the common audiophile assumption that virtually every speaker sounds different after some extended break in period.

As for the other mechanical devices, I'm not sufficiently expert myself to rule it out, but do you have any measurements showing changes in the signal after time? 

If it's based only on the "I Swear I Heard A Difference" method of vetting such "break in" that's not too compelling.  I've seen audiophiles literally claim everything breaks in sonically, even their AV racks!  

And remember OldHvyMec was making claims about cables.

@tonywinga 

Amir says, "When it comes to non-linear distortions, audiophiles are notoriously poor at hearing those artifacts.  It is for this reason that even poor measuring gear is praised as sounding good."

Tsk Tsk, another generalization without supporting data- bad science.  Actually, everyone is poor at hearing non-linear distortions because they occur naturally around us and even in our heads, inside our ears to be specific.  That is one reason tube amps without negative feedback sound better but SS amps without negative feedback can sound good too but look worse on paper.  

You complain about my statement not being scientific and general and proceed to give me the very definition of those in your response!  :)

I am happy to back my statement with proper research and references.  To hear small impairments you need to know what to listen for.  And for that, you need to understand the underlying system.  Audiophiles tend to be poor at both even though some walk around thinking they are very gifted on that front.

For above, reason, when we care about reliable data, we use trained listeners.  Earlier I showed research by Dr. Sean Olive on reliability of different groups of listeners when testing speakers:

Notice how poorly audio reviewers did which audiophiles tend to regard to have superior ability to evaluate other gear.  Harman research showed that you need to have 10X more trials or number of testers to create the same set of reliable data as their trained listeners.

Trained listeners are extensively used in other domains such as hearing compression artifacts. When at Microsoft, and without that knowledge initially, I suggested to my manager of signal processing group that we recruit the hundreds of audiophiles we had at the company to identify impairments in codec. Blind test was created and distributed to them.  A while later my manager came back telling me how poorly they had done.  And that they were essentially no better than general public, and far worse than our trained listeners.  I asked him to give me an example.  He gave me one of the tests where I easily found the artifact.  I apologized for wasting his time and from then on, we continued to use our trained listeners (of which I was one).

It took me about 6 months of intensive training to learn to find small non-linear artifacts.  Those skills now allow me to hear them in broad set of tests which most audiophiles would not dare to take let alone pass.  I gave an example of this in video I post on blind testing (I think).

Back to your comment, I have tested a ton of tube gear.  I find their distortion to either not be audible or simply manifest in brightness, lack of clarify and edginess.  Yet audiophiles make the claims you repeat.  There is not one publish controlled test which backs their or your position.  None.  So if you are a fan of "science," I suggest not repeating folklore like that which can't be proven. At least not on the same breath as telling me I was unscientific.

@whipsaw 

Do you have evidence supporting the claim that amps which measure better than tube (or Pass Labs) amps are preferred by a high percentage of audiophiles because they are more true to the source?

No one has done such side by side testing.  If anyone should do that, is Pass Labs given the huge premium they charge for their amplifiers. If they sound better, then it  should be trivial to perform controlled tests to show that.  Alas, not only do they not provide such a listening test, no do any other amplifier companies.  So much for "it is the sound that matters."  In really, it is the marketing that matters.

To be clear, I have not test any Pass Lab commercial product. I did test his "ACA" DIY amplifier design and thought it was a distortion factory.  I am confident I can put together a test that shows it to perform very poorly against some other amplifiers without all of its flaws.

@amir_asr  given the huge premium they charge for their amplifiers.

How is price part of audio science? 

@amir_asr 

No one has done such side by side testing.  If anyone should do that, is Pass Labs given the huge premium they charge for their amplifiers. If they sound better, then it  should be trivial to perform controlled tests to show that.  Alas, not only do they not provide such a listening test, no do any other amplifier companies.  So much for "it is the sound that matters."  In really, it is the marketing that matters.

C'mon, Amir. As you know, Pass Labs amps have been very well received by audiophiles over decades now. They clearly have sound signatures that are pleasing to the ears of many listeners, and the suggestion that a meaningful percentage of those reactions would likely change if only those listeners were to A/B their amps with those that measure with less distortion is dubious, at best. And the same could be said of high-quality tube amplifiers.

To be clear, I don't doubt that some listeners would arrive at conclusions that would be at odds with their long-standing, stated preferences. But given the vast weight of the feedback from audiophiles who apparently prefer amps which measure with some distortion in the audible frequencies, it is, in my view, highly improbable that their choices are primarily due to marketing-related biases.

there's a pass int-150 for 3k on tmr right now...very tempted to do my own "testing" 

...and I guess my next question to Amir would be: Do you have any significant data to show the correlation of SQ of components priced from $1000 to $30000 as an example. 

Amir, 

You are getting emotional.  I was pointing out that it is not just the subset of audiophiles that struggle to hear nonlinear distortion but everyone is that way.  

I have this image of King Kong hanging on the top of the Empire State Building as the biplanes circle overhead machine guns blazing.  The A'gon'rs being the biplanes.  You being the big gorilla.  Or did I have to clarify that?

Perhaps you can buy everyone dinner and we all go our ways.  You are clearly not listening to what we all are saying and we are not drinking your kool aid.

@russ69 +1

I think measuring amp by driving pure resistive load is not enough. It is much harder for amp to drive actual, far from ideal, non-linear, distorting. etc speaker, vs resistive linear load. Many PASS amps on my opinion are good choice for driving high end speakers. 

@prof

Right, and after OldHvyMec made his first major statement, there was a dispute, then 2 pancake posts and then posts where he could peddle his wire and break-in assumptions.

As to cartridge break-in, just admit you know nothing concerning vinyl playback. There are no tests that I have viewed indicating new versus 50 hour break-in results in sonic characteristic changes. HOWEVER, just because it wasn’t tested, I am not going out on a limb by stating nearly every other site discussing and reviewing cartridges ALWAYS recommends listening/reviewing after break-in, 15 hours or 50 hours or whatever.

I will swear that I have heard break-in of my new cartridges after 50 hours of play, which has happened over a dozen times since I’ve owned VPI turntables (40 years in 2023).

As to speakers, my dynamic speakers took about 50 to 100 hours to go from very dark sounding to open sounding.  Not subtle, very obvious.  I'm with OldHvyMec 100% on mechanical break-in. 

As a cable beta tester, I hear the cable raw, then burn it in for 24 hours.  I can't say I can always tell if it sounds better, but I can always tell when it sounds worse.  Depends on the cable.  Doesn't matter to ASR.   Tubes, maybe an hour or two.

Some of the greatest sounding musical instruments, in fact most all musical instruments  and even concert halls were designed and built without computers and electronic analysis equipment.  They were designed and built by artisans with skilled hands and ears.  It's all about what we hear.  When someone tries telling me I am not hearing what I think I am hearing, well that goes over with me about like putting a tax on a child's piggy bank.

@tonywinga 100%. I stated as much concerning modern (100% computer/AI designed) concert halls versus classic old concert halls.  Funny thing is that after the initial performances of a season or two, nearly all of those modern halls are then "renovated," "adjusted" to sound better generally determined by listening to the results and finding fault.  

Luckily, my audiophile friends, mastering engineers and audio equipment manufacturer don't doubt what I hear.  The former and latter friends have provided me with acoustic room suggestions in my former home.  The mastering engineers just share music.

@tonywinga you have for all intents and purposes said you are not honest with yourself. Every day our brain presents us with ideas that are not true. Many have inflated view of aspects of themselves, while having deflated views of other aspects. Our brain lies to us all the time. But this time, this time while listening to audio, it is 100% fallible? 

As to cartridge break-in, just admit you know nothing concerning vinyl playback.

Well, no...I won’t say I "know nothing", thank you very much. I will say I’m far from an expert and try not to make strong claims that aren't backed up with good evidence.

But, that’s me.

Am I to just fall to my knees and accept your claims as received wisdom? Or is it ok with you if I keep my critical faculties engaged?

You followed up with a bunch of claims about break in, all based on what you claim to hear. Perhaps you did hear break in. But then audiophiles believe they can hear everything under the sun - imagination and listener bias is, sorry, a fact of life and I see no reason to pretend it isn’t a variable. So, again, given I have seen many people with technical knowledge refute claims about break in, if an audiophile is going to claim break in I’ll wait for stronger evidence, in the form of measurements (or blind listening tests). You can do you, I can do me.

I had never heard of ASR until I saw a post about Synergistic Research cable and snake oil. I found my prior purchase of a Foundation digital cable really bad in my system. I was offered a trial of their 3rd from the top. I tried it and was blown away at how superior it made my digital system sound. So, I followed the link and wound up with the CD trimmer comment. I made my comment and was pilloried. Further comments came with character assassinations, innuendo and just perverting my initial neutral statements concerning the trimmer, SR cable and whatever. I got mad but mentioned my recording history and work with studios. Nope, just got worse.

Despite being blocked on my Chrome IP, I easily access the site now which is why i know about current postings. I am no dummy, I started using computers full time in 1982 (I am not inferring that anyone who is not computer literate or a late comer to computers is a dummy, just that I can write programming for the software that I used in the past).

I was really mad when I began this forum. Nearly all long time Audiogon posters on this forum see what ASR is about and how angry they are at us for using our hearing ability to determine what is most pleasing in the reproduction of music. Measurements are important. Lack of measurements or less than optimal measurements does not condemn a product. Good listening determines if the result is favorable or not.

Okay, how many vinyl listening Audiogoners claim that cartridges break-in or don’t?

@prof denies what he doesn’t hear or know. Yes, if you can’t hear a cartridge break-in on a good sounding system, you could be suffering with a bad short/long term sonic memory, inadequate vinyl playback system, etc. I’m at a loss here. I have never met someone who does not believe that cartridge’s break-in with use and change their sonic characteristics and usually set-up requirements. I’ve spoken to many cartridge distributors over 50 years and they all said wait until the cartridge break-in to hear what it can do.

I want to know what profession or professor of @prof is as his handle indicates?

fleschler

How do you determine the truth of claims, in your method?

Let's say a manufacturer makes claims about an expensive new digital cable.

Your ultimate method of adjudicating that claim, as far as I can see, is whether you listen and  hear a difference from another cable, or not.

So let's say Audiophile "A" listens and says "I hear an improvement in the sound" and audiophile "B" listens to the SAME set up and concludes "sorry, there is no difference in the sound."

Who is right?  They've both used precisely the same method yet arrived at contrary conclusions.

Is the audiophile who claims to hear a difference ALWAYS in the right?

Dang it guys. Any electromechanical device (loudspeakers/phono cartridges) will break-in. How long does that take? Beats me. Then we have electronic components. They need to burn-in to reach their final electrical state. How long does that take? Ask the manufacturer. And finally, we have cables. Do they need to be burned-in? I don't know but if somebody says it took X hours for a new cable to sound right, I have no reason not to believe them regardless of what the mechanism might be, even if it's their brain adjusting to the new sound. So, what's the problem that needs to be disputed? 

@whipsaw

 

C’mon, Amir. As you know, Pass Labs amps have been very well received by audiophiles over decades now. They clearly have sound signatures that are pleasing to the ears of many listeners, and the suggestion that a meaningful percentage of those reactions would likely change if only those listeners were to A/B their amps with those that measure with less distortion is dubious, at best. And the same could be said of high-quality tube amplifiers.

To be clear, I don’t doubt that some listeners would arrive at conclusions that would be at odds with their long-standing, stated preferences. But given the vast weight of the feedback from audiophiles who apparently prefer amps which measure with some distortion in the audible frequencies, it is, in my view, highly improbable that their choices are primarily due to marketing-related biases.

I think ASR is—to a degree—caught in its own hermetic logic here.

One the one hand, Amir does ignore Toole’s advice that speakers should be listened to comparatively for evaluation to be meaningful. Even something as simple as setting up a curtain and turntable and enlisting helpers from his cohort of followers is dismissed. Instead he does measurements first, preps some PEQ filters based on them and on his experience, then compares "blind" original and EQ’d playback through the device under test (just one speaker). Nothing wrong with most of this (except, I would listen and take notes first—as another Kippel user, Erin of Erin’s Audio Corner does—that argument has run a few times on ASR). But I can AB between EQ and original and hear differences (most can) and that doesn’t substitute for comparative testing of (as Toole recommends) three or four speakers. Apart from the magnitude argument, it’s unclear how one type of listening is kosher and another isn’t, but that’s a longer discussion and requires more than ventilation of talking points.

But when it comes to amplifiers, ASR rarely listens, with the general justification that magnitude of differences are too small to differentiate (with some exceptions). As you can see in this thread all listening (that isn’t done by ASR with partial protocols) is routinely dismissed as sighted bias, expectation bias, focus bias etc (you too can play bias Whac-A-Mole). This despite the fact that double-blind ABX isn’t logistically straightforward. And dismissing well-defined subjective listening protocols like those published by B&K or B&O, or the expert listening we use daily to produce music. The escape clause is of course the null test, but that's also a longer conversation.

In cases where people do make a good-faith attempt to try DBT, it can also degenerate into a shambolic series of gotchas. Before my time at ASR I nonetheless read a long and winding thread where enthusiast/reviewer GoldenSound tried to respond to a challenge Amir made to compare DACs. Amir managed to misunderstand straightforward logical concepts like one vs many wrt comparison criteria, using that to reset his conditions halfway and eventually doxxed and banned his antagonist, and possibly some other posters in that conversation. I modified my appreciation of ASR ethics based on that thread. We are all flawed human beings though, so I enjoy ASR for what it offers and tolerate the weaknesses.

@axo1989

Thank you for your thoughtful response. I second your final sentence, in particular.

@russ69 

How is price part of audio science? 

I am not in audio science.  I am posting here.  Ask a pedantic question, you receive an answer in kind.  :)

@axo1989 

But when it comes to amplifiers, ASR rarely listens, with the general justification that magnitude of differences are too small to differentiate (with some exceptions). As you can see in this thread all listening (that isn’t done by ASR with partial protocols) is routinely dismissed as sighted bias, expectation bias, focus bias etc (you too can play bias Whac-A-Mole). 

The pushback like that is made when the claimed sighted tests go counter to solid body of engineering and research.  Say a power cable improves the sound because you swapped one for the other and proof of being right is "I have been an audiophile for 30 years" and you rightly get strong pushback.  Don't offer such as proof point and you are generally fine.  And even if folks object, you should be cool because you weren't going to prove anything.

How else do you want us to behave?  I once asked my doctor if he could help me with research into weaknesses of blind testing.  He just about threw me out of his office!  He said, "Amir, the foundation of what I do is based on blind testing; I can't participate in any attempt to cast doubt on it."  It wasn't my intent to cast doubt but I fully understood his position and continue to see him.

Before funding ASR, I was the co-founder of Whatsbestforum (WBF).  We thought by allowing both camps to state their position, life would be good.  Well, it turned out to be anything but.  The conflict eventually crept between me and my partner and I sold out my shares and got out.  I decided then to go the "pure" route and start ASR.  The name clearly states that we are committed to teachings of audio research and engineering for decades.  We don't pretend to be smart enough to invent our own rules of universe for audio and champion that to everybody else with vengeance as folks are doing here.

Net, net the response you mentioned is what you should expect if you come and make outlandish claims. You have seen me respond similarly here.  As I said, it is a jazz club and you shouldn't expect folks to take kindly to you demanding that you play country music.

You want to come and challenge our position? Do so with solid research and science driven listening tests.  Be ready to defend yourself and not cry victim with "oh they ask for controlled test and tell me about bias."  Of course we do.

Indeed, many people who have a short life in ASR mistakenly assume they are stating something new to us that we are just going to roll over and accept.  Member @kota1 for example shows up and says every cable needs to be broken in for 100 hours or the test is invalid.  We have heard these claims a million times.  Don't be the million and one member who thinks you should just throw that at us and we go, "oh, I didn't know that; thank you for that information!" 

Read the forum a bit and get educated on what and who we are.  Then participate if you need to.  You are welcome to challenge us on every topic.  Members do that to me all the time.  But be ready for heaven's sake with some back up worth more than a fortune cookie paper!

@axo1989 

One the one hand, Amir does ignore Toole’s advice that speakers should be listened to comparatively for evaluation to be meaningful. Even something as simple as setting up a curtain and turntable and enlisting helpers from his cohort of followers is dismissed.

This was a planned activity from day one that I started to test speakers.  Indeed, i have held on to a mountain of speakers for this very purpose.  But you may have heard of a thing called the pandemic.  Our local audiophile group where I was hoping to conduct such tests stopped meeting (and went virtual) so the project is on hold.  Meanwhile, one member did post such a comparison: 

And a much more sophisticated one using a turntable was created as well:

I highly encourage such efforts.  I provided speakers for the second phase above to the organizer and happy to do so for anyone who likes to conduct them.

Such testing is extremely time consuming.  But good news is that anyone can do it.  You don't need my experience or instrumentation.  So no one should be waiting on me for it.  My time is best spent providing objective data such as measurements.

To position this as me against Dr. Toole's teachings is very much out of line.  Nothing remotely is true in that regard. I simply don't have the resources or time to do this kind of testing on every speaker that lands here.  

My listening tests in reviews is provided on "as is" basis. I do them because if I didn't, I would get more complaints.  "Oh, he doesn't listen."  I have tried to make more sense out of them by developing the EQ technique.  The outcome there has been quite positive with many trying my EQ profiles and liking them over stock performance.  If folks want to ignore them -- and many do -- it is no skin of my nose.  I perform them because I am curious myself how the measurements translate into sound and a form of listening training.

@amir_asr  "I am not in audio science.  I am posting here.  Ask a pedantic question, you receive an answer in kind.  :)"

You have been very generous with your time, I'm sorry I am testing your patience, but I've am seriously curious how you inject price or value judgements into your reviews.

@tonywinga 

Some of the greatest sounding musical instruments, in fact most all musical instruments  and even concert halls were designed and built without computers and electronic analysis equipment.  They were designed and built by artisans with skilled hands and ears.  It's all about what we hear.  When someone tries telling me I am not hearing what I think I am hearing, well that goes over with me about like putting a tax on a child's piggy bank.

Once more:  listening tests are the gold standard in audio research.  No one is telling you to substitute measurements for it.  

What we say is that don't go believing marketing claims that have no verification with controlled testing, or make sense at engineering level.  We prove the latter with measurements.  Company claims the power conditioner lowers your audio system noise?  Well, we measure that.  If the result is that noise has not changed one bit, then you know the claim was wrong. 

Why is this so odd for the few of you to accept?  You say your local water is making you sick?  Folks come out and measure to see what is in it.  If it is pure and clean, then that is very important information. 

Importantly, don't confuse creation of art with replay of it.  Our business is the latter. The two are completely different universes.  Audio equipment should NOT be in the business of creating or modifying art.  If it is, then it is not high fidelity.  And will impart the same signature on every music you play -- something I dislike dearly.

As to what you think you are hearing, that is NOT in doubt.  What is in doubt is what you say it means when you did not block all other senses than your ear.

My listening tests in reviews is provided on "as is" basis. I do them because if I didn't, I would get more complaints.  "Oh, he doesn't listen."  I have tried to make more sense out of them by developing the EQ technique.  The outcome there has been quite positive with many trying my EQ profiles and liking them over stock performance.  If folks want to ignore them -- and many do -- it is no skin of my nose.  I perform them because I am curious myself how the measurements translate into sound and a form of listening training.

That all makes sense to me.  Given the range of audiophile viewpoints you can't please everyone.  If you don't listen, you'll get complaints about that.  If you do listen, you'll get complaints that you aren't using a rigorous enough protocol.  If you use a rigorous blind protocol, you'll get pushback from the anti-blind-test faction who think blind tests obscure results and you should have listened "in relaxed sighted conditions, like a normal audiophile."

Personally, I think that yeah, blind listening to speakers would hue most consistently to the ASR remit.  But your compromise of  "here are my impressions take them or leave them" seems a reasonable compromise.

@amir_asr

To position this as me against Dr. Toole’s teachings is very much out of line. Nothing remotely is true in that regard. I simply don’t have the resources or time to do this kind of testing on every speaker that lands here.

I’m not positioning you as "against" Toole, simply stating that you ignore that specific recommendation. Ignore in the sense of "intentionally disregard" which is what you’ve specifically stated here. If the semantics don’t suit you, change ignore to "doesn’t follow".

Your business model is based on rapid testing and fast turnover and of course that has advantages and disadvantages. As this thread started by conflating ASR with Darko, let’s consider that he takes some weeks to listen to and test a speaker. Now—leaving aside potential flouncing about objectivist vs subjectivist approaches, relative value and the like, please—you could take that time, but you choose not to. Perfectly valid decision (and you’d test far fewer speakers). But it’s your turnover preference that precludes turntable testing, or other comparative methods. So ASR has both value, and weaknesses. We appear to agree.

I highly encourage such efforts. I provided speakers for the second phase above to the organizer and happy to do so for anyone who likes to conduct them.

The turntable will be interesting (albeit just a handful of posts so far, and no progress since March?)

I once asked my doctor if he could help me with research into weaknesses of blind testing. He just about threw me out of his office!

I’m not sure about the value of your anecdote about your doctor. I’d prefer you addressed my question about your allegations against my earlier post, for example.

Finally, thanks for the invitation to post at ASR, I’d never thought of that. 🙄😂😉

relevant to the topic at hand (as well as the initial confusion between darko and ASR outlooks) - here john darko interviews paul barton of psb, who explains the importance of measurements and listening tests (following toole’s methodology) to his process, as well as the subjective choices he has to make as he finalizes his designs. fascinating stuff, confirms much of what both "camps" are saying:

@russ69 

You have been very generous with your time, I'm sorry I am testing your patience, but I've am seriously curious how you inject price or value judgements into your reviews.

Thank you.  As a general rule, I do not incorporate price.  If a piece of equipment performs superbly, I give it my highest recommendation irrespective of price.

There are situations where I bend the rule.  For example, in amplifiers that cost less than $100, I allow impairments that I would not in more expensive gear.

Conversely, if something is super expense and performs middle of the road or lower, it will get my scorn.

All in all, it is a personal opinion which can be discarded and reliable measurements and other data used for yourself.  At the risk of stating the obvious, measurements don't care what something costs.

@axo1989 

I’m not positioning you as "against" Toole, simply stating that you ignore that specific recommendation. Ignore in the sense of "intentionally disregard" which is what you’ve specifically stated here. If the semantics don’t suit you, change ignore to "doesn’t follow".

It was a spin and a debating stunt which I called you on.  Dr. Toole's collective research is how a specific set of measurements highly predict listener preference.  Those measurements are created by me in every speaker review (and then some).  This is the lead and core of my review.  

My listening tests and EQ are a supplement that I have chosen to include.  A ton of people have argued against it on ASR.  I have answered them in the video I produced.  It is not important or core to my review of a speaker although I personally value it.

What you did was elevate the listening test to something it is not, then complain that it doesn't follow the extensive protocol Dr. Toole used for research.  That was improper and I responded to you as such.

Your business model is based on rapid testing and fast turnover and of course that has advantages and disadvantages. 

Another debating stunt.  I do not run a "business" to have a model.  I have a hobby which creates great value for large swath of audiophiles and the audio industry in general. That hobby is based on objective data on audio gear and explaining the science and engineering of audio.  

As an engineer, I try to optimize for the resources I have.  A $100,000 speaker measurement system needs to be in constant use to provide that level of value.  Me sitting on a single speaker to test for weeks and months doesn't provide the right value.  Creating predictive measurements absolutely does.

And it is not like you have shown any of those editors that hold on to gear perform comparative blind testing of speakers.  They have the time according to you but waste it away with who knows what.  You want to complain about something, complain about that.

@fleschler 

As a cable beta tester, I hear the cable raw, then burn it in for 24 hours.  I can't say I can always tell if it sounds better, but I can always tell when it sounds worse.  Depends on the cable. Doesn't matter to ASR. 

Now do the same thing blind, run a camera and repeat 10 times.  Let us know if you can tell the raw from burned in cable.  Should be easy for you to run such a test.  I post the video on how to do it.  All of us at ASR would love to see such an experiment.  Your anecdotal claim above where you included your eyes and full knowledge of what is being tested, not such much.

@amir_asr " At the risk of stating the obvious, measurements don’t care what something costs."       

Thanks for the reply. BTW: I do look at the measurements you post on your website and read the forums occasionally. I don’t think I ever joined though. I could see that my experience would not be valued so I don’t look at the website as much as I look at others. If the environment allowed my input, I would be much more likely to use the website more. I don’t want to make a wrong guess but I’m thinking if this thread was on ASR, I’d be long gone. Thanx again for joining in the discussion, I’ll lay back now and watch on the sidelines. No need to respond, you have been very patient with me. Thanx.

Another story. :)

Our audiophile society was invited to a stereophile reviewer's home to see and experience his setup (he just left that organization).  He had a new amp to review against his own.  Room was too small for all of us to go in there so we divided into two groups.  First group went in and came out.  Without telling us anything, we went in there.  At the end, reviewer asked which amplifier sounded better which folks did.

When we came out to join the larger group, we realized the first group had voted the exact opposite!!!  The reviewer said he had played the amps in reverse order for them vs us.

You see how faulty sighted evaluation is?  

We create controls in listening tests to create reliable outcomes and avoid the above.

Having done various blind tests over the years, it's a very powerful lesson.  It's too bad many audiophiles haven't experienced their 'sighted' impressions dissolving away when they can't use their knowledge of which piece of gear is actually playing.   There's nothing that sinks in like an actual experience.

(And many audiophiles see blind testing as almost synonymous with "detecting no sonic differences."  Where in fact plenty of differences have been detected in blind tests for various things.  That is of course one way various codecs were arrived at.  And I've had some positive results for identifying differences in my own blind tests).

Of all the comments I have read in this thread, this by djones astounded me:

“This is wrong, as a measurement guy "better sound " is subjective and not anything I pay attention to. “

I thought I misinterpreted the comment, but then

No idea if it is better or worse? Seriously? This explains the reply I got on ASR when I asked “What does it sound like”. I was told that doesn’t matter because it measures perfectly. Well let us take one example. I listen to a lot of equipment and have done many comparisons together with friends, at shows and also in demonstrations by companies. My major complaint about a dac that ASR rates as the best measuring dec is that it sounds like crap. I stated that the mid range is shrill, particularly when dealing with female voices. I was categorically told that I was wrong. I am not wrong – for me it sounds shrill. I was thrown off the site for this.

I would not use this dac if it were given to me gratis. Fine, I accept that others may like it, however I do not. Interestingly only one of the 8 that was present at the comparison thought it was satisfactory. We did not know what the brand was at the time. Perhaps the fact that we are all interested in classical music was a deciding factor.

Another example is an amp that ASR wets themselves over as the best measuring amplifier of all time; it is actually quite good, especially for the price, but there are far better amplifiers for my taste.

Regarding Amir’s comments here. Frankly I find them offensive. He has come along with the attitude of "Let me educate you." I do not need him or anyone else to educate me on what I like and dislike. Nor to I need to be lectured that I do not understand the Science. There are many people far more experienced and qualified than Amir of whom I prefer to take note.

@amir_asr

It was a spin and a debating stunt which I called you on.

The part I asked you to respond to was your statement that my "takeaway" after amplifier listening was contrary to audio (and psycho-acoustic) research. The points I re-iterated were quite sound, but I was interested in your counterpoint (as opposed to your talking points).

What you did was elevate the listening test to something it is not, then complain that it doesn’t follow the extensive protocol Dr. Toole used for research. That was improper and I responded to you as such.

Relax mate, I’m not trying to trick you with wicked (sorry, "improper") sophistry. Nor am I running a comprehensive analysis of ASR test methodology, with or without "elevation". No need for all the mansplaining.

The simple point of my earlier post is that we can listen to gear, with some experience and awareness of the pitfalls, without always following the strictest of protocols and still glean meaningful information. I described doing it upthread, you do it yourself with loudspeakers (and you say you think it has value).

Another debating stunt. I do not run a "business" to have a model.

Is there anything to be gained from this semantic quibbling? I’m sure you know that in English "business" has several meanings, including "an activity one is engaged in" and isn’t restricted to commercial activity. If you don’t like the semantics of "business model" just think of it as "modus operandi" or "general approach". You know what I mean. Argue the substance.

And it is not like you have shown any of those editors that hold on to gear perform comparative blind testing of speakers. They have the time according to you but waste it away with who knows what. You want to complain about something, complain about that.

Pure whataboutism. How about those reviewers, eh? Come on.

Perhaps this link is something all people who posted on this site should look at,
 

@laoman @russ69 Thank you for your experiences which reveal more about Amir and his buddies who accompanied him to Audiogon to TELL us what we need to make intelligent equipment decisions.  Just like you laoman, I don't need someone to educate me at 66 after my extensive listening experiences, what sounds best to me. 

When I upgraded my digital cable two months ago, I've had audiophiles and non-audiophiles just revel in the digital reproduced music.  The audiophiles say "best ever sound."  Well, I can't say that because I have heard absolutely fantastic sound systems better than mine (and expensive).   No problem as I'm living within my means.  My wife, a very tough customer who says she is now always concentrates on the sound before listening to the music much to her consternation, just sat and listened for several hours (also unusual) to a variety of smooth jazz and her rock music (1970s to heavy metal) when I upgraded the cable.  She said it sounds like vinyl.   

Today, Positive Feedback has an article by Roger Skoff-Hi-Fi Weather? Roger Skoff Writes About Something You May Not Have Thought Of…

I found the end of the article not dealing with weather’s affect on sound propagation most enlightening (especially if true) "None of those changes is massive but, with even average human hearing having a 100 decibel range from the lowest sound we can hear to the loudest sound we can bear without injury, our ears have a single-scale resolution range of 1 to 1 BILLION—far greater than any known test instrument—so even differences that might seem prohibitively small may be clearly audible."

What’s Best Forum I have for several years joined and enjoyed camaraderie of other audiophiles forums. That site is generally more congenial and well mannered than any other forum site I frequent. Disagreements do not involve character assassination or defamation. (Except with posters experiences with Acoustic Fields company/owner). I prefer hearing about others experiences than reading about the gold standard measurements as Amir prefers. Good for him but Darko uses different measurements and often reveals different answers. Choices and variable methods to conduct experiments occur in measurements. So Amir is not the greatest tester of audio equipment. There are serious drawbacks on measurements themselves. If measurements are the gold standard, then listening in a system in a room is the platinum standard.

Audioshark, Audiokarma and older sites I visited were quite nice and amiable. DIYers also was very informative lacking acrimonious postings. Why is that?

ASR Frgirard Your points are

The willful ignorance of the primacy of speaker-room coupling by taking refuge in the magic of equalization.

the willful ignorance of the primacy of the decay in the bass and the rest of the spectrum.

the spending of thousands of dollars on equipment to listen in rotten rooms, a total lack of rationality here and on AG.

I spent $150K+ building my listening room from the ground up (12" 3000 psi steel reinforced slab, 16" multi-dimensional walls with chambered activated carbon bass trap filters, etc). I use no equalization. I have various room treatments for mid and high frequency absorption, smoothing, diffusion and decay/reverb. In doing so, I do not require state of the art equipment to obtain a very high quality sound reproduction absent state of the art/expensive high end equipment. Putting well designed speakers of moderate price (even $1K) will react superbly in my new room. I’ve built several other sound rooms in prior homes but they were good but not great. Then again, I have heard a few superb large systems in very large untreated rooms at audio shows at high cost ($1m+). The room is generally very important (as it is in my live recordings) but sometimes the equipment can perform well in a wide variety of settings (Von Schweikerts).
 

I personally think the ASR mob are looking to ’justify’ spending much less money to get a hi-end sound.

They lean on ASR for reassurance that a Topping DAC and £500 amplifier is all they need to spend in order to get the very best sound possible.

More than likely most of them do not have deep wallets so perhaps somewhat ’envy’ those that can spend the money to - in reality - get far better sound than measured perfection can really give you.

I do not have a deep wallet to enjoy hi-end at its best - so I could easily been seen as one being suckered into believing that measured sound is the best sound to save money. That would suit me just fine - if only that were true.

Good post chopandchange. You are probably accurate. I also think Amir has fled. He has closed the thread on ASR which was intended as an insulting rebuttal to this thread.
It is a pity he has gone. I was curious as to how he would answer the comment that his testing methodology on the Chord M scaler was faulted. He consigned a whole product to the dustbin based on faulty methodology.

@fleschler 

"None of those changes is massive but, with even average human hearing having a 100 decibel range from the lowest sound we can hear to the loudest sound we can bear without injury, our ears have a single-scale resolution range of 1 to 1 BILLION—far greater than any known test instrument—so even differences that might seem prohibitively small may be clearly audible."

Point 1:  I encourage you to learn enough about audio, science, or math to check the potential validity of what you wrote.  Point 2:  When I have the music turned up loud and I am engrossed in the music, my wife needs to shout to get my attention. I just quickly Googled that normal talking is about 60db. That means I am struggling to hear 60db. Hopefully that means something to you as it relates to what you wrote.

Perhaps it is my background in a science related field, but I find these concepts rather simple to understand, but without that understanding, I could see myself susceptible to believing things that are not true. Couple that with a belief you have nothing to learn and you become susceptible to both external suggestion and internal misdirection.

Good for him but Darko uses different measurements and often reveals different answers. Choices and variable methods to conduct experiments occur in measurements.

I looked through Darko's site. It was obviously not the first time I was there. I failed to find one measurement except in his discussion of bass decay. Are you referring to his ears as measurement devices?

@laoman ,

Regarding Amir’s comments here. Frankly I find them offensive. He has come along with the attitude of "Let me educate you." I do not need him or anyone else to educate me on what I like and dislike. Nor to I need to be lectured that I do not understand the Science. There are many people far more experienced and qualified than Amir of whom I prefer to take note.

Can you tell us these people who are far more experienced and qualified? I am serious, I would appreciate a list.

I don't think Amir is at all lecturing anyone on what they like or don't like. That would be obtuse and Amir does not appear to be obtuse. Do you need to be lectured on the science?  To enjoy music? No.  To continue to contribute in a valuable fashion on audio forums without being incensed by what others have written and misquoting what they have said?  I think for many on this forum, that should be self evident. For many, when they read something on a forum that does not agree with their world view their first inclination is to think, "you are wrong, wrong, wrong". Maybe we all do that. It is the next step that defines you. You can get on that forum and say you are wrong, wrong wrong, and likely be the one that is wrong, or you can open up Google, and start learning. Proving yourself wrong is much less contentious than someone else doing it, and it is ultimately more satisfying.

@chopandchange , your post is nothing but insult attempting to look like high reasoning and deep insight. It is a tired trope and best left inside your head and not put on a forum. I could as easily say that the posters on this forum are incensed with Amir because he has shown, with hard to refute measurements, that you can achieve perfect recreation of an audio signal with a Topping DAC and a £500 amplifier.

Amir is trying to prove a negative. That has never before been accomplished.  
Nothing to see here. 

I’m running out of popcorn and the mudwrestling match shows no signs of a clear winner in sight. 

@laoman ,

Props for that link to the Darko exchange with Amir. It spoke volumes. Darko is such a class act. I could (and should) learn a thing or two from him on how to engage the absolutists and dogmatics one encounters along the way.

All the best,
Nonoise

The problem with measurements above all, is the logistics. Take loudspeakers for example. We can't drag a loudspeaker home and run a Klippel sudo-anechoic measurement on them before we buy them (even if that system could provide useful information on panel loudspeakers). So, even the measurement guys do what we all do. They go down to the shop and listen to the speaker and buy the one they like. 

the assumption that there is a fundamentally rational, measurable basis for emotional responses to an experience (like listening to music) is flawed. no one denies that there is ample research and data to back up assertions about engineering, and everyone is aware that there are results from controlled listening tests that indicate the preferences of the majority (of test participants anyway). my view is that figuring out "what sounds good" is something different. life is not a controlled listening experiment, and our individual experiences with music are shaped by circumstances which cannot be generalized, let alone replicated. 

the act of measuring gear is not a threat or a problem for anybody. i agree with amir that more information about a product is always a good thing. where i disagree is when people insist that i shouldn't trust my senses when i am determining whether a piece of equipment is good for my system.  reviewers with a subjective bent don't bother me at all, for the same reason - seems blandly obvious that what sounds good in john darko's room might sound like junk in mine. how do i figure it out? i buy the thing and return (or sell it) if it sucks!

i've done this with topping gear which measures great on ASR. some people love topping - i don't say they're wrong or have bad hearing or only care about specs. my $100 schiit dac simply sounded better and the topping gear was ultimately not worth the expense. why? i don't know! and at the end of the day, i don't care. will i spend another thousand or two bucks trying to better the schiit? probably. i like trying new things and seeing if i can hear the differences - it's fun

I’m running out of popcorn and the mudwrestling match shows no signs of a clear winner in sight. 

We all won. Everybody got to express their opinion and some readers may have learned a few things or at least been given some food for thought. ASR closed their thread while we are still up and running. I'm not too proud of the personal attacks but very happy this thread stayed open despite the contentiousness. 

I could (and should) learn a thing or two from him on how to engage the absolutists and dogmatics one encounters along the way.

The amount of "projection" in this thread is really something ;-)