Best practices when conducting a DAC comparison

Since the question will be asked:

Denafrips Terminator

Schiit Yggdrasil (my current primary DAC)

Exogal Comet Plus

and if I have the time and energy an Oppo 105 and a Wadia 121.

With Roon it is easy, just play the same track to two grouped zones (two DACs) and then switch inputs on a preamp to hear either one. Just make sure you level match with test tone.

It is not an easy thing to do, because the differences will be tiny at most. The methodology is simple, however. First, look at measured test data to find if there any weird anomalies. Second, if you are then still inclined to do listening tests, make sure that levels are very very precisely matched (to witin 0.2 dB). The best way to do so is by using a good volt meter. Next, switching should be near instant, given humans' short audio memory of at best a few seconds. Finally, comparison should be blind, to avoid expectation bias, and should be repeated multiple times, to exclude random error.
It is the unwilingness to submit to such a rigorous test protocol that generates the audiophile fairy tales of night and day differences between DACs and other electronics. People hear differences and believe that what they hear is real. But that is not necessarily true. Good luck - it is a hard job.
I can also predict that those not trained in scientific research methodology will immediately pounce on me.

@shadorne Thank you. I have a preamp that will allow for that and I will be using Roon / Tidal.

What I don’t have is the extra isolation platform and footers so that variable is equal.

I’ll do my best to keep levels matched. I know it’s tricky.

I'll also have to get an extra set of matched interconnects.

@willemj   Thank you for your response.

My methodology will not be that rigorous, but I understand and appreciate the points you are making. I'll try to do my best.

I recently compared 6 USB cables and yes, it isn't easy. But I learned a lot, despite the difficulty and tediousness of the process.

Thanks for the reponse. I know it is not easy, and it is precisely for that reason that I deplore the gratuitous praise for some components over others. We are, of course, largely let down by the audio press that mostly only publishes advertorials. They should measure with the expensive audio analyzers that individuals cannot afford to buy, do the double blind listening tests, and help us avoid spending money to upgrade from already perfect components.
I do remember participating in a blind listening tests of amplifiers by Quad's Peter Walker. At the time people were arguing that his new 405 amplifier was not as good as his 303, and that his valve amplifier was actually the best he had ever done. So he had this rig where you could compare without knowing what you were listening to. I thought I could identify them, and he greatly enjoyed demonstrating to me that I had only been guessing randomly. A more elaborate version was published here: http://www.keith-snook.info/wireless-world-magazine/Wireless-World-1978/Valves%20versus%20Transistor...

The thing is, getting to know an audio system takes time. Hearing its strengths and weaknesses is a process that requires time and a multitude of different recordings. Walking up to an unknown system and trying to guess what is what is bound to fail. 

It's funny, when I was into modding my car, these same conversations took place amongst motorheads, with some going by feel and impressions and others demanding dyno numbers..

@david_ten

Well, you may or may not know that I just went through a grueling and somewhat meticulous exercise myself when I received my Yggy.
https://forum.audiogon.com/discussions/new-yggdrasil-first-and-second-impressions

Grueling because my angst was sort of self imposed in that I had a short return window to work within.

Anyway, long story short, I truly believe I conducted proper comparative testing.

- I used the same (type and length) RCA cables connected to two different source inputs.

- I ensured the volume control on my amp never changed.

- I enlisted the help of another listener.

- I sampled many types, varieties, etc. of music.

- I sampled hi and lo rez formats, but only PCM.

- I tried (and helper) both blind and non-blind testing

- I used a playback source that allowed me to "loop" music (5-20 seconds would suffice)

- Settled on particular music that highlighted the differences. Namely, Elegant Gypsy by American jazz fusion guitarist Al Di Meola. If you can get your hands/ears on that recording, do so. Virtually every second of music on it can be used effectively to A/B among DACS.

- I consulted with other "audio-experts" here on the forum and in a private club I’m associated with, in order to better understand whatever specific nuance I should attempt to listen for.

- I conducted testing at different times of the day, with and without alcohol.

- I conducted testing with equipment cold (initial startup)

- I conducted testing with equipment warm (2 hrs plus on)

- I conducted testing at the PLP in my listening environment AND other locations throughout the room. BTW, many users report how incredibly "live" the Yggy sounds even *in another room* and I cannot agree more. 

- I did other things too, just can’t remember right now :)

I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter.
Substantively the take home is, of course, that a great effort was needed to maybe, but only maybe, establish a tiny difference.

I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter.
Substantively the take home is, of course, that a great effort was needed to maybe, but only maybe, establish a tiny difference.

Not according to the output voltage specs of the *three* components used (Yggy, ERC-3, UDP-205). But I respect your (good) point.

Look, I admit, I do not run or have access to a "scientifically exactly perfect audio laboratory".

For purposes of what I was trying to achieve, accomplish, etc. I’ll stand pat on my decision that in my case the Yggy was and is the correct purchase, and that my testing is *more than* satisfactory.

EDIT:

I agree differences in digital can be tiny and hard to discern. But it’s worth the effort (IMO) once you realize, understand and can prove with reasonable certainty that the component change was/is worthwhile. I wouldn’t go back to delta-sigma from multi-bit R2R. No way. *BIG* difference. (with the Yggy that is. My assumption is that multi-bit implementation will best delta-sigma in any case). More "authentic sounding", period.

For the record, you did a great job. I don't have any measuring gear at home either, and I similarly try to make sense of my imperfect observations. However, as a research scientist I try to stick to proper protocol to shield myself from all kinds of known biases. Sometimes I hear differences, but sometimes I really don't, even on my pretty revealing Quad electrostats.

@willemj 

Thanks!

@david_ten 

Something else I did that I forgot to mention in my earlier post herein was keep a written log of the results. Seriously. I wanted/needed to "keep score" because often will be the case - especially if when trying blind testing/comparison - that in one instance you favor device A and in another you favor device B. Do that a hundred times and it could be worthwhile to reconcile the results from a written log, in addition to whatever other methods you use.

@toddverrone

The thing is, getting to know an audio system takes time.

Agreed.

I'm planning to take my time, especially with the newest component.

The A / B comparisons will be at the very end of the process.

@gdhal 

I have read and kept up with your thread. I also posted a few times.

Luckily, I don't have to worry about the return window since I purchased outright because I believe in what @toddverrone mentions in his post. I do not want to rush this. 

Thank you for the detailed response and guidance. Since you recently went through this, the list of all the steps you took is terrific and very helpful.

I'm not sure about the cold start one for me, but I understand why you did it when you explained it on your thread.

@willemj 

My methodology will (likely) not be at a level where you would consider it to be valid. If my business depended on such matters, I'd be much more thorough and particular.

So white coat it will not be. Perhaps somewhere between white coat and cigar jacket. : )

@audioengr 

Steve, great points across the board. I'll do my best to check off each one. Thank you!

.... I do not want to rush this. Thank you for the detailed response and guidance. Since you recently went through this, the list of all the steps you took is terrific and very helpful.....

@david_ten

You’re welcome, David.

I agree with you (and @toddverrone). I didn’t want to "rush" anything either. Thing is, at least in my case (and this is usually the case with any new component), the manufacturer may provide an "audition window of time" whereby you can return the piece if not satisfied. In the case of the Yggy this is 15 days. Would I have preferred 30 or more, sure! Point is I only "rushed" in order to maximize the return opportunity. Once you know for certain you are keeping your unit, at that point absolutely, take all the time you need/want to compare it to other pieces. Presumably you would be living with your gear for many years at least, so there is certainly plenty of time to "check" (and even re-check) it during that time.

Best of luck David. You are rather thorough, knowledgeable, etc. so I have good reason to believe you will make the right decision in any case.

The Denafrips Terminator was delivered early yesterday morning and went in system yesterday afternoon.

I've ordered the Gen 5 USB board from Schiit and will be doing the install myself.

The Denafrips Terminator was delivered early yesterday morning and went in system yesterday afternoon.

I’ve ordered the Gen 5 USB board from Schiit and will be doing the install myself.

I don’t understand. I thought the Schiit Gen 5 is an upgrade for any Bifrost, Gungnir, or Yggdrasil with the Gen 1, 2, or 3 USB inputs.

Your Denafrips Terminator (I assume this one http://www.denafrips.com/terminator.html at face value looks like a monster DAC and it should be at more than twice the cost of the Yggy. The website indicates it uses an Amanero USB Interface, which does appear to be of similar physical size to the Gen 5, but can you just swap the USB cards out?

@gdhal  I'm not sure I understand your question. Yes, the Gen 5 board upgrade applies to all three Schiit DACs you list.

My Yggdrasil is about 2 years old and has the Gen 3 board in it. I've been holding off on the upgrade until I had another DAC on hand.

@david_ten 

I initially thought, because of the way I interpreted your 11-07-2017 8:24am post, that you were wanting to install or use the Gen 5 board with or within your Denafrips Terminator. I also assumed (because you wrote that you have the "Schiit Yggdrasil (my current primary DAC)" that *is already* Gen 5, not knowing your current Schiit Yggy has the Gen 3. This is/was my misunderstanding. Thanks for the clarification. 

@david_ten 

So may I assume that it is your intention to compare **on a per input basis** the Schiit Yggdrasil to the Denafrips Terminator ?

What inputs will you be comparing?

Will you be posting your observation results herein?

I would also ensure all the components are broken in, especially the Terminator DAC.

@gdhal 

Got it. No, I don't want to mess with Mr. Zhao's flagship work!. : )

My main output is USB from a Sonore Signature Rendu SE.

I'll primarily be focused on comparing the USB input between all DACs.

@muzikmann  Yes, especially the Terminator. The manufacturer advises 300-500 hours; and yes I know how some feel about this. : )

As @toddverrone mentions in his post, spending time with the components will also help accomplish this.

I will start with the critical listening for the Terminator towards the end of November. I'll energize and run signal through it 24/7 until that time.

I'll also keep the Yggy and the Comet energized. 

You are certainly being thorough.. I'm interested to hear what you find. Keep us posted!

Thanks, Todd. I think you will appreciate what follows.

My main concern is that my current system has been (mostly) optimized with the Schiit Yggdrasil within it. I struggle with this very issue in the comparisons and component evaluations I read, whether professionally done or owner based.

I’m especially concerned about cabling.

I recently completed a USB cable comparison. It was difficult yet very educational. I’m well aware that the cable I found to perform best to my preferences and goals may not match as well or deliver the same outcome with the Denafrips DAC (again to my preferences).

For example, I chose the Cardas Clear Reflection XLR ICs for my headphone system. They are performing very well in the 2ch system, but the choice was made for a different application and goals. Would another brand of cable or moving up the Cardas line-up be a better fit for my 2ch system, with the Yggdrasil in it? For me, the answer is likely.

The ’million’ dollar question (for me):

........ is using the same cabling across the different DACs the best way forward?

OR

........ is it better to compare optimized systems, with the components standing on their own within them?

I believe the second approach has a greater degree of fairness in terms of determining endpoint differences and performance preferences. It, of course, is not nearly as practical or financially feasible.

[Note: At a minimum optimizing around the power and signal cables in and out of the DAC is a somewhat more viable option]

I realize we have a number of Audiogoners who find none to minor differences between cabling. I respect this. Further, keeping the cables identical will make the process straightforward and much, much easier for me.

For those of you that do find cabling an important piece, I’d love to hear your approach to this. Thoughts? Thanks.

The best measure for me to demonstrate the quality of the low end is the Beatles "Come Together" test. The first break where Paul let's his Bass "A" string linger out ("Joker do what he please"). Even if generally listening you cannot tell the differences between dacs, amps, cables....this one little passage will give it all away. You'll know it when you hit a sweet spot because you can feel growl of the bass string. You can switch components and notice that something is missing. This isolated bass snippet is copied and pasted a couple of times in the Beatles LOVE soundtrack as well.

........ is using the same cabling across the different DACs the best way forward?

OR

........ is it better to compare optimized systems, with the components standing on their own within them?

Both ways is technically best from a comparison and arrival at conclusion(s) perspective.

@david_ten,
I am rooting for the Terminator because I have been following it since its conception, it has been tempting my wallet for some time now.

How can somebody say that there will only be tiny differences between dacs? Maybe this guy hasn’t heard a lot of dacs. There can be big differences between dacs. Also, I don’t agree that you have to use the same cable or input to compare different dacs. If you are only using usb, then sure you have no choice, but if you have an option of using i2s, toslink, or spdif, then I would use the cable that the dac prefers because you want the best sounding dac that you can afford. I would never go back to usb, and I would tend to use i2s if possible.
Also, ignore all the measurement specs that some reviewers post, IMO they are worthless. Who cares if 1 dac has .0001% distortion and the other has .0005%, you think you are going to hear that? That’s like asking the Porsche dealer to give you a readout of the compression of each cylinder before you buy a new Porsche. You have the best instruments to do your evaluation and that your ears.

What I haven’t seen is your requirements for the dac and this is where you can get huge differences. Do you want the dac to support MQA? DSD and which resolution? I have some MQA tracks that surpass vinyl by not a small margin.
I would make sure that your volume levels are set the same and then test each dac with your favorite music using various inputs unless you are stuck with only 1 interface.

@steakster  Agreed! They definitely make a difference, including the upgraded DC cables. I've been running HDPlex to power the switch, server, etc. for quite some time now. I'll probably play around with different brands once the major components are finalized.

there will only be tiny differences between dacs relative to most speakers, room tmts., etc.

galvanic isolation can be a biggie for a DAC -- besides an opto-isolation technique, a LPS, and/or star-quad DC power cables are good ideas

@rbstehno Thank you for your response.

Some differences: the Denafrips offers DSD; higher resolutions of PCM; Over Sampling; three I2S inputs, and an extra set of Coax, TOSLink, AES each, but no BNC --- over the Yggdrasil.

Unfortunately, probably fortunately for some : ) MQA is not an option for either DAC at this time. It would have been a nice feature to have.

I am not clear on the consumer benefits of MQA.  It does seem DRM capable however...

@randy-11 I’ll be installing the Gen 5 USB board which does offer galvanic isolation for the Yggdrasil. I believe there is galvanic isolation built into the Denafrips (but I need to double check this). For now, optical isolation is off the table; I am aware of the benefits.

Consumer benefits of MQA? Have you actually listened to MQA and compared this to other mediums or are you listening to other posters remarks? Again, I don’t listen to what other people say about new technologies because there are many times they are slanted. Some people are still pissed off from the sacd/dvd-audio debacle and won’t consider any new medium, or people way over the top about every new technology. The best thing to do is hear it for yourself using your own equipment in your own room. My buddy had 5 different dacs on loan to him so he could evaluate them with his own equipment, in his room, and using his music. This is the best way to evaluate a piece of equipment.

A recent blind test with a large group of listeners showed no statistically significant differences with 24/96 pcm. I fact, if anything, there was a slight preference for the 24/96 pcm. The test methodology was not perfect, but I have seen nothing better thus far.

@aberyclark   Thanks for the tip on The Beatles 'Come Together' track. I'll give it a critical listen.

@gdhal   Thanks for your ongoing thoughts and advice.

if you don't hear a significant difference in the first 30 seconds then keep what you have!  Well that is my opinion anyway.  Happy Listening.

So if a recent blind test indicated that piece ‘A’ is the best, would you go out and buy this without hearing it 1st? I don’t give a s$&# if a group or a reviewer indicates this it that, I would have to listen to it 1st before spending a dime on it.
if my ears hear a difference between pcm versions, or DSD, flac/aiff, or MQA, then I will pursue that format.

Personally, I let the reviewers do the heavy lifting.  I read all of the reviews I could find on the DAC I was interested in and paid attention to the associated equipment they were using.  I was interested in the Auralic Vega nad purchased it based on the reviews of several seasoned audio/digital audio reviewers and decided to purchase the DAC based on what their observations were.  I believe I made an excellent choice.  I live in Cleveland and when I purchased the Vega, there were not many dealers that were well versed (or equipped) with digital audio gear.  Just my 2 cents.

Dan

I agree that the only way to compare audio equipment is a/b any contenders.  I also agree that matching volume if allowed two inputs is critical.  Is there any better way to hear the difference?

There are so many very good brands out there, reviews and comments/opinions  like these provide a guide in narrowing down selections within your budget...

jhv

@bigkidz   Is your 30 second test after break-in and when doing a quick A / B? Thanks.

@rbstehno  I agree. We are the ones living with our gear, usually for significant periods of time. These are very personal choices based on our experience and preferences and system synergy.

@dpetri  To date, to my knowledge there is only one major review of the Denafrips Terminator DAC that I purchased. I had made the decision to do so prior to the 6Moons review, but it came out approximately the same time I placed the order.

Certainly preferable would be a number of sources, owners and reviewers, to help with the choice. Good as a guide, especially if one finds their preferences line up with a reviewer's. But as @rbstehno  posted, ultimately it's our ears, and system, and room, etc. etc.

@dumacker  Thanks. I guess that is the crux of the issue: "Is there any better way to hear the difference?"

As a general update, I have two weeks of run time on the new DAC. I still need to install the Gen 5 board in the Yggdrasil and get it running as well. Everything is still on track for an evaluation between both towards the end of November.

Please add any other thoughts on the topic and approaches on how to best go about a DAC comparison. Thanks.

david_ten I build Direct Heated Triode DACs - most people never heard a DHT product.  First you should have the material that you are extremely familiar with so you know what you are looking for so when you swap out the product you are using, you should know if the newer product makes enough of a significant change based on your listening experience.  I know that many people will say that it has to be in the system for some time for you to get to really hear what the new product offers, well they are probably right, but IMO only, I usually get to hear what it does within 30 seconds probably because I know the material extremely well.  for example, I have a switch in my preamp where I can change resistors by turning a rotary switch.  There are 5 resistors that you can select from.  My partner could hear the changes without knowing which resistor was which simply because we have listened extensively to those resistors in the past, so he was able to know which resistor was what immediately.

I wrote the 30 second test because I manufacturer Direct Heated Tube products so I am biased.  Typically I bring my DAC over a persons home to audition or ship one to them.  So the DAC is usually already a broken-in model.  Most people want to hear it in their system before purchasing so we offer a 14 day in-home trail.  Once it is plugged in, if you don't hear an immediate change to the sound of your system or want the DAC offers within 30 seconds, send it back.  After about 20 to 30 minutes the tubes begin to warm up and the sound will change again for the better.

Happy Listening.  To the rest of you reading this, keep the comments to a minimum!

 


 

david_ten
Best practices when conducting a DAC comparison

Each dac will have a different gain structure (volume), don’t trust your ears to level match.
It must be done with a 1khz sine wave and measure the ac mV (millivolt) at the speaker terminal.

Cheers George