2. pc-audio
  3. 139819-120414

Where are the blind streaming quality tests?

I've been searching around for awhile now trying to find a good article/report on how many audiophiles on a decent system (at least $1000) can hear the difference between Apple Music and TIDAL and can reliably pick which one is "better sounding" in a blind test environment. It seems most blind tests I have found show that people really can't say one service sounds better than the others. But they often are using a <$100 audio setup. Many people on here claim that they can hear a huge difference between AAC and FLAC or MQA etc. but without any evidence or test results.

So I'll challenge everyone: get someone else to switch between lossless audio and compressed without your knowledge. Also have them sometimes just do a "fake switch" where they say they changed something but really kept playing the same thing. Or better yet use the NPR test online. Can you still reliably pick the lossless audio over multiple trials with a variety of music? Post which songs you used, if you guessed right on that track, and maybe a list of gear you used for your test. Let's see if we can get some real scientific results here!
Ag insider logo xs@2xmattlathrop
Ag insider logo xs@2x
cleeds
6,004 posts
 
Conducting a scientifically valid double-blind listening test involves much more than just having a buddy "switch between lossless audio and compressed without your knowledge." And if the test doesn't follow scientific protocols, its results are of no value at all.

Have you ever participated in a real double-blind listening test? It's r-e-a-l-l-y tedious, even if you're just a test subject. Organizing, conducting and tabulating the results of such a test are even more work. That's part of why you don't see more blind testing by hobbyists, outside of the scientific and commercial domains.


n80
2,133 posts
 
Double-blinded randomized tests of this sort need not be overly complicated. Yes, they will be tedious. Yes, the design would need to be just right. Yes, the stats can be a little mind numbing. But it would not be expensive or even time consuming.

I think there are other reasons these tests are not done: audio gear vendors and retailers do not want it done. It is so much easier for them to have a high tech, high cost niche in which virtually every important characteristic is seen as and accepted to be subjective.

It is a bit strange to me, as a new audiophile, that this type of testing is conspicuously absent even as I see many people asking for it.

In my other hobbies (photography, for example) there is much more rigorous testing and reporting, even in magazines/sites that stand to lose advertising, than in the audiophile industry.


Ag insider logo xs@2x
cleeds
6,004 posts
 
n80
Double-blinded randomized tests of this sort need not be overly complicated. Yes, they will be tedious. Yes, the design would need to be just right. Yes, the stats can be a little mind numbing. But it would not be expensive or even time consuming ...
If you’re testing a cross-section of listeners - which is necessary to have sufficient sample size - it is very time consuming. Each listener must be accorded multiple trials of whatever duration they need. So if only because of the time factor, it is indeed an expensive undertaking.

It is a bit strange to me, as a new audiophile, that this type of testing is conspicuously absent even as I see many people asking for it.
Everyone who asks for such a test is free to conduct such a test. Why don’t they? Perhaps because it’s time-consuming. Tedious. Challenging to do properly. And - when you’re done - the results are often vague, or inconclusive.

Ag insider logo xs@2x
mattlathrop OP
26 posts
 
@cleeds 
First off I have conducted and participated in psychological studies relating to human perception. So I do get what is truly needed. But a simple blind test as I have described would be an excellent start given that no one seems to have even done that. Sure the results won't be getting published in any journals, but it would be a good jumping off point for a more formal study. 

the results are often vague, or inconclusive.
This statement is exactly why I want people to do this. My hypothesis is the same as yours. I think people will find that the results aren't a clear cut answer. BUT if you were an audiophile starting out and read these forums you would think that if you don't listen TIDAL you have just wasted your money on an expensive stereo (this *literally* happened to me when a dealer who will remain nameless told me he didn't want to work with me if I only listened to Apple Music...)

Just to share my personal experience: I feel I can tell the difference between TIDAL and Apple Music when I switch back and forth, but when I took the NPR test with some Audeze headphones I didn't do better than randomly guessing. 

Ag insider logo xs@2x
mattlathrop OP
26 posts
 
@n80 It is neat to find another audiophile who also likes photography! I completely agree with your point about photography. Imagine choosing to buy camera A or B based on seeing a set of test images taken with both cameras where you have to look at black for 2 seconds every time you flip between them.
More to discover