Why HiFi Gear Measurements Are Misleading (yes ASR talking to you…)

@cleeds

Prof: They view their own opinions on how to evaluate gear as the default - e.g. "The Only Way To Truly Evaluate Gear Is By Listening To It, Like We Do" ...

Hey @prof that’s the logical fallacy known as the strawman argument. In fact, I’ve never seen anyone here make that claim, yet you put quotes around it.

You know very well the point I was making. Have you even forgotten the whole start of this thread?

Stated in the very OP of this thread:

"Takeaway - trust your senses! Science doesn’t always measure the right things, in the right ways, to fully explain what we are sensing. Therefore your sensory input comes first. You can try to figure out the science later."

This is clearly the Start With A Trust In Your Senses approach I’ve been talking about. And it has been echoed in various ways through the thread by others!

The idea is "start with what we believe we hear." That’s the bedrock. And then we can perhaps search for measurements to explain what we hear, but the point is First Trust The Hearing...and if we don’t find measurements support what we hear, well then it’s a problem with the measurements...we’ll have to wait for science/measurements to catch up to What We Can Hear.

This privileging of "Listening" over measurements has been echoed throughout this thread. And I’m sure you know it.

It seems to be your mission to convince others that they are wrong. That appears to be the root of the problem here and explains your increasing frustration.

I’m waiting for the day that you ever produce anything but a strawman of what I write, or my view. Your track record of misrepresenting me, or not responding to my actual arguments, borders on heroic.

(And of course all your argumentative replies don’t count as "trying to show why someone is wrong." How convenient).

@tsacremento 

@amir_asr, I have a few questions regarding measuring equipment. Genuine questions, not “poking the bear” nonsense. 

Appreciate the constructive tone.  :)

1. Can unit-to-unit variations significantly affect measurement outcomes?

I don't think so.  These devices are not mechanical as to require assembly line alignments and such.  We (I and membership) have done some spot checks have confirmed that random units purchased matches my test results.  There have been a few exceptions.  I once tested a Schiit pre-amp (?) where one channel had 10 dB lower SINAD than another.  Schiit reached out to me and offered another unit without that deviation.  I guess in a perfect world we would buy another unit on our dime and re-test.  But I took Schiit's word that the unit selected was picked at random.

This concern used to come up a lot in the past.  When it heated up a lot, I had Topping reach out to me saying they would pay for me to buy every product of theirs I had ever tested to see if the performance was different.  I have a lot of respect for their ethics in this regard and chose not to do that. But their offer stands.

As to part variations, yes, this exists.  Fortunately for most components, the actual value is not critical.  A 1000 microfarad power supply filter will do its job whether it is 20% lower or higher in value.  In a few places, this matters a lot and there, companies pick high precision 1% parts and such.  In addition, we have design techniques such as feedback which eliminate a lot of variations from output of the audio device.

Finally, there are other people making measurements similar to mine now.  A fellow in China who goes by the alias L7Wofl for example, reviews chinese gear, testing other samples than me, with results that correlate excellently with mine.

Net, net, I don't think this is a factor to be worried about especially in the context of large variations in offerings from different companies.  Other answers below.

@thespeakerdude

That is because we do know how to translate measurements into what is heard and it is far less variable.

You can’t tell how a speaker sounds simply by how it measures. There is 0 possibility or we would all by buying speakers from mail order catalogs of graphs. More proof you don’t have a system, and are making stuff up ad lib.

@tsacremento 

2. Does a unit’s chain-of custody or provenance impact your confidence in the results of a unit’s test results being truly representative of the model?

For the vast majority of cases, no. Most of the reasoning was explained in my last response.  Let's remember that if there is an issue with Golden samples, it would affect other reviewers far more than me because they exclusively get samples from manufacturers.  In my case, a large number of products for test come from members.  A good portion of these are purchased new and drop shipped to me.  And large percentage of used ones are current products.  I occasionally test vintage products or discontinued ones because they are available on the cheap on used market.  Manufacturers are welcome to challenge the results of any used product tested but I have yet to encounter one.

@tsacremento 

3. How audible will the difference between an electronic component producing 0.1% THD and one producing 0.001% THD be when played through a transducer generating between 1.0% and 2.0% THD?

The nature of distortion is different in electronics vs speakers.  For example, speakers have no self-noise (passive ones anyway) whereas electronics do.  This is why you can put your ear next to a tweeter and hear hiss and buzz.

On pure distortion front, speakers produce their most distortion in bass region where we are not critical anyway.  An amplifier distorting will do so at all frequencies.  Have an amp clip and you can hear it on any speaker even though it may not reach the bass distortion of said speaker.

There are stated of the art speakers and headphones that have clocked 80+ dB SINAD which is the limit of what I can measure for them. 

That said, detecting non-linear distortion is not easy for most listeners. So you could say that maybe people can't hear even elevated distortions 1 to 2%.

Here is the thing though: the only reason a piece of electronic generates 0.1% THD is due to sloppy or bad ideas in design.  It is almost never the case that it is done to make the equipment cheaper.  Indeed, by far, the situation is the reverse: you pay far more money for a gear with more distortion and noise!  You pay more to get more noise and distortion.   Yet you can buy a device from companies that care that have provably inaudible noise and distortion for very reasonable cost.  We learn about this by measuring.  If we didn't, we would be going by marketing words of expensive gears and not objective, reliable data.