Audio Science Review = "The better the measurement, the better the sound" philosophy

Well you would not expect consistent unanimous results from blind listening tests involving different listeners  because different people have different sonic preferences.But there is value in what was found to be good on average as that product is most likely to sound good to most people-or at least most experienced listeners because that is what the listeners were[usually a mix of industry designer/engineers and audio journalists].

And Hi Fi Choice did use more than one listening session per product.And all products were level matched.The difference with other unsighted tests being that the room was familiar to the listeners  and I consider that would make a difference in terms of being able to hear more differences between components and something that should be included in any blind listening tests but isn't.

"Sting isn't going to personally travel to your house and perform while you're eating your nightly bowl of spaghetti." 

@kokakolia - This actually makes me very happy 👍 

Reliability and consistency are big issues that I think are overlooked by all reviews, somewhat out of practical necessity. To really do it right is quite expensive and time consuming. Multiple units should be tested over long periods of time with a variety of different inputs and outputs and configurations. My personal experience with digital audio and video equipment is that intermittent problems can and do pop up that are audible and visible without totally destroying the signal. The "bits is bits" argument clearly isn’t true. The music or video can come through distorted at times but mostly intact, meaning some bits are being misrepresented. After a system reset the problem usually resolves for the short term. If the effect is subtle it might be misattributed to a fundamental flaw in the quality of the media or in the playback potential of the component in question, when really it’s just an intermittent inconsistency caused by various devices not coordinating well with each other.

As for audibility of various cables and other electronic components, what might shine some new light on those and how they interact with each other would be a set of measurements off a speaker in an anechoic setting rather than test bench measurements of the components in isolation. It would be important to keep everything identical in the measurement space each time, only changing equipment upstream. I suspect that some unexpected differences in the speaker’s output might be measured as amplifiers, dacs, cables, etc. are interchanged. To keep things exactly the same with each measurement the speaker would probably have to be built into a wall so that it’s front side was in the anechoic measuring space and all other equipment was outside, preventing any potential effects from the size and shape of the components changing the response in the measuring area. Even that might be a bit misleading because some equipment might be effected by being exposed to the sound the speaker is making in a sort of feedback loop. So a second speaker might have to play in the equipment room to provide that input, but it would be very important that none of that sound leak back into the measuring area. You can see why this sort of testing isn’t done very often.

A scientifically established fact on limits of human perception can easily be misapplied. Take for instance the claim that a light blinking fast enough will appear to be not blinking to the human eye that’s looking straight at it. This is true and demonstrable, but someone might say something like "If it’s blinking at a certain speed you can’t detect with your eye that it’s blinking." That’s not true, because as you sweep your vision across you do detect that it is blinking. So there’s a fact that’s been well established and then there’s a story about that fact that isn’t really true. I don’t know that I can come up with a really good comparison with audio but one might be masking effects. MP3 data compression relies on known tonal masking effects in human perception to throw away data that you "can’t hear" anyway. This may work better in some situations than others. I recall listening to a recording of the cocktail party effect, where on the recording you can hear a whole bunch of voices talking at once but can’t really make out a word anyone is saying because they all mask each other. In stereo over speakers I still couldn’t make out a conversation. Because it was, I believe, a binaural recording using a dummy head, when listened to over headphones or using a divider plate between my speakers to prevent crosstalk, I suddenly could make out a nearby conversation because spacial cues became available to more distinctly separate it in distance from the surrounding hubbub. So why am I bringing this up? I’m just wanting to acknowledge that while I really appreciate reviews with measurements like ASR does, I do acknowledge that there’s a strong possibility that there are things being missed that are audible. I think it's a worthy effort to keep trying to find ways to measure everything we perceive and enjoy in audio. With more understanding we have a better chance of more consistently getting the perception that we want.

asctim

... while I really appreciate reviews with measurements like ASR does, I do acknowledge that there’s a strong possibility that there are things being missed that are audible ...

That's especially true because ASR doesn't bother to listen to everything it measures. Without correlation to what we hear, the measurements don't have much value.