Best practices when conducting a DAC comparison

The thing is, getting to know an audio system takes time. Hearing its strengths and weaknesses is a process that requires time and a multitude of different recordings. Walking up to an unknown system and trying to guess what is what is bound to fail. 

It's funny, when I was into modding my car, these same conversations took place amongst motorheads, with some going by feel and impressions and others demanding dyno numbers..

@david_ten

Well, you may or may not know that I just went through a grueling and somewhat meticulous exercise myself when I received my Yggy.
https://forum.audiogon.com/discussions/new-yggdrasil-first-and-second-impressions

Grueling because my angst was sort of self imposed in that I had a short return window to work within.

Anyway, long story short, I truly believe I conducted proper comparative testing.

- I used the same (type and length) RCA cables connected to two different source inputs.

- I ensured the volume control on my amp never changed.

- I enlisted the help of another listener.

- I sampled many types, varieties, etc. of music.

- I sampled hi and lo rez formats, but only PCM.

- I tried (and helper) both blind and non-blind testing

- I used a playback source that allowed me to "loop" music (5-20 seconds would suffice)

- Settled on particular music that highlighted the differences. Namely, Elegant Gypsy by American jazz fusion guitarist Al Di Meola. If you can get your hands/ears on that recording, do so. Virtually every second of music on it can be used effectively to A/B among DACS.

- I consulted with other "audio-experts" here on the forum and in a private club I’m associated with, in order to better understand whatever specific nuance I should attempt to listen for.

- I conducted testing at different times of the day, with and without alcohol.

- I conducted testing with equipment cold (initial startup)

- I conducted testing with equipment warm (2 hrs plus on)

- I conducted testing at the PLP in my listening environment AND other locations throughout the room. BTW, many users report how incredibly "live" the Yggy sounds even *in another room* and I cannot agree more. 

- I did other things too, just can’t remember right now :)

I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter.
Substantively the take home is, of course, that a great effort was needed to maybe, but only maybe, establish a tiny difference.

I respect the genuine effort to do this properly. Methodologically there was one error, however. You may not have changed the volume control, but the output voltage of the two units may have been different. Mind you, I am not saying that this is what made the difference, but methodologically it does matter.
Substantively the take home is, of course, that a great effort was needed to maybe, but only maybe, establish a tiny difference.

Not according to the output voltage specs of the *three* components used (Yggy, ERC-3, UDP-205). But I respect your (good) point.

Look, I admit, I do not run or have access to a "scientifically exactly perfect audio laboratory".

For purposes of what I was trying to achieve, accomplish, etc. I’ll stand pat on my decision that in my case the Yggy was and is the correct purchase, and that my testing is *more than* satisfactory.

EDIT:

I agree differences in digital can be tiny and hard to discern. But it’s worth the effort (IMO) once you realize, understand and can prove with reasonable certainty that the component change was/is worthwhile. I wouldn’t go back to delta-sigma from multi-bit R2R. No way. *BIG* difference. (with the Yggy that is. My assumption is that multi-bit implementation will best delta-sigma in any case). More "authentic sounding", period.

For the record, you did a great job. I don't have any measuring gear at home either, and I similarly try to make sense of my imperfect observations. However, as a research scientist I try to stick to proper protocol to shield myself from all kinds of known biases. Sometimes I hear differences, but sometimes I really don't, even on my pretty revealing Quad electrostats.