Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

@amir_asr , thanks for responding with a constructive perspective. It would be nice if all debates on the internet were this friendly and constructive, but it is what it is, as they say. Another way of saying it isn’t what it should be.

The problem with your argument is, I agree with everything you have said. I’ve many times been tricked by my own mind, only to realize later that my initial findings were flawed. But over time, and not a heck of a lot of time, say 2-3 weeks of “living with” a component, I find that I can arrive at a more stable and true subjective assessment of a component. One that personally, I find goes beyond what the measurements can tell us.

I don’t feel the need to win arguments, and I am comfortable with alternate perspectives coexisting with my own. I believe Audio Science Review is great for checking out how gear physically performs, but my own perspective is that too much emphasis is attributed to these measurements in terms of how much we can expect to subjectively enjoy a particular component. I think this is the fundamental point where our perspectives diverge.

I believe that other factors come into play as well. One particular example that seemed to rub some the wrong way is ASR’s review of the Mustec MA005, where ASR reported poor performance. I also see forum posts from some folks who report that the Musetec bests the more expensive Mola Mola Tambaqui which performed well in ASR’s measurements. Including one guy who sold his Tambaqui after he heard the Mustec. However, in each of these posts from people who prefer the Mustec, I’ve noticed they are feeding the DAC with an inexpensive Roon Rock NUC or a Mac Mini. I do wonder if the tables would be turned if feeding the DAC’s with a higher end streamer like the Grimm MU1. In other words, I wonder if the Tambaqui in these cases is simply exposing the flaws of the source more easily than the Musetec, or in other cases, the Holo May (which some say bests the Tambaqui) which yes, shows that it rolls off the upper frequencies more sharply that the Tambaqui. All this to say, that sometimes, maybe a lesser performing component may sound better given lesser surrounding components in one’s system. Which again, judging a piece by measurements would never provide any useful context in these situations.

Why then SS amp always has a haze where as tube amps always has a transparent sound but we all know tube always have inferior frequency response vs. ss.

In short you can't measure it.

 

I have never heard a haze when listening to an SS amp, or some startling clarity with tubes. However, one factor that may affect your personal listening experience is most tube amplifiers have high output resistance. Couple that with the right (wrong) speakers and you can accentuate some frequencies by even a few db. Maybe you interpret that as haze, maybe it even compensates for a speaker/room response issue.  Then again, it could just be what Amir stated a biased view.

 

 

 

I have never heard a haze when listening to an SS amp,

Likewise.  I've done plenty of listening to SS and tube amps, never heard this SS "haze" of which he spoke.

Some people project their own impressions as some objective truth, as if just claiming something means it's true and "has to be explained."

Amir showing up here is like Richard Dawkins dropping in to a Church revival ;-)

 

 

 

 

@nyev

 

But over time, and not a heck of a lot of time, say 2-3 weeks of “living with” a component, I find that I can arrive at a more stable and true subjective assessment of a component. One that personally, I find goes beyond what the measurements can tell us.

 

The thing is, if you are starting with a flawed methodology - in your case sighted listening - using the same flawed methodology over a longer time doesn’t yield more reliable results. Biases can modify, or settle in over time, and you may be attributing this to the gear rather than changes in your perception, which has been Amir’s point.

The thing is most of us really don’t want to be told...or learn...our perception isn’t reliable. We stake so much on it. But, reality doesn’t bend to our desires, human foibles are what they are.

Now in talking about blind testing like this the hackles will often go up "hold on, so you are telling us we can’t rely at all on our perception? Now we have to blind test everything or we can’t have an opinion or do this hobby?"

No. Not exactly.

First of all, clearly our perception IS relatively reliable. We get through the day using our senses. And blind testing is not easy, and depending on the gear can be utterly impractical. I don’t think any audiophile HAS to engage in blind testing or pay attention to audio science. We can all do whatever we want.

HOWEVER, the fact is we have biases and they are relevant. If we don’t acknowledge that variable, we are simply being ignorant.

So given blind testing (and often, measuring) is impractical for many audiophiles, how to navigate these problems? One way is to just say "look, I don’t care about measurements or blind testing, I’ll go just on what I seem to hear." Fine and dandy for anyone who wants to do that. But IF you are going to take your impressions and make claims to other audiophiles based on those impressions, especially if they are dubious claims in light of generally accepted technical theory and practice, then you shouldn’t be surprised if some audiophiles hold a skeptical opinion and point out they would prefer more rigorous data - e.g. measurements/listening tests controlling for biases - before they accept such a claim about said piece of gear.

So nobody has to do blind testing to conclude what they want, but IF we want to be more careful about conclusions - learning from what the scientific method has to tell us - THEN it makes sense to look for evidence that is less susceptible to run of the mill sighted bias effects.

And, though blind testing may be inconvenient for most of us, there is plenty of engineering, testing, science out there ALREADY done by competent people, that we can look to when trying to evaluate a claim about equipment.

So, the way I navigate the problem of the impracticality of blind testing everything, vs recognizing the variable of human bias is basically a heuristic like ’Extraordinary Claims Require Extraordinary Evidence.’

So, if someone tells me they auditioned some Wilson Audio speakers and some MBL speakers and they describe the differences they heard, I’m perfectly happy to provisionally accept they heard those differences. Sure, could be some bias involved, but it’s also extremely likely they heard real sonic characteristics in each.

That there are audible differences between speakers is well established by theory, practice, experience, measurable evidence and even listening tests controlled for sighted biases.

That’s not the case for some of the claims in high end audio, though. The claim that an expensive USB or AC cable will likely alter the sound from any cheap (competently designed) cable IS quite technically controversial. Many knowledgeable people (the ones not selling such stuff) will explain why this his highly unlikely given how those things work. So...for THAT type of claim I personally will raise the bar for the evidence. I’ve had the personal experience of "hearing obvious sonic differences" between cheap AC and high end AC cables, but when I performed blind tests where I didn’t know which was being used, none of the sonic differences were there at all and my guesses were random.

So when an audiophile swears up and down he heard something "so obvious" when changing a USB or AC cable, I’ve learned that we really can have very strong but erroneous impressions (something science has told us for a long time).

It won’t matter if the anecdotes pile up, because they are all using the same method that allows for sighted bias. I’ll wait until I see measurements showing actual changes in the audio signal and/or people able to reliably pass blind tests for choosing between such cables.

Neither you, nor anyone else here, needs to have the same criteria I do.  Follow your bliss.   But there ARE good reasons for having such criteria and it shouldn’t be seen as some sort of heresy for which the skeptical person is made a villain, just by giving this reasoning.

 

Cheers.

 

 

 

 

 

@prof , like Amir’s points, you make rational points as well.  But really what we are debating is, which approach is more flawed - our senses because our biases get in the way, or measurements because while they reflect “reality”, they do not reflect on how we perceive a component’s physical performance.  For me, I believe the latter is far more flawed.  It doesn’t mean that the flaws you are pointing out in subject assessments are not valid - they totally are!  But IMO it’s the best we have, barring the day we can figure out  measure human perception of physical stimuli.  Once we can do that, we are into Bladerunner territory!