Eljaro - No argument here when it comes to your ears, eyes (yes aesthetics matter) and your wallet being the ultimate judge of an audio component. I think this is well understood... but perhaps not well practiced (myself included) ;-)
I am sure portions of the following have been published in print and online. Pardon the lengthy approach but your post provides a great opportunity to discuss the importance of software selection.
So when it comes to our use of only two pieces of well recorded (minimally mic'd, uncompressed, no effects applied, careful mic placement, etc.) acoustic pieces to compare the equipment, I agree that a broader representation of music would be more ideal. However, the logistics of the event simply prevented the introduction of one more recording. As simple as this would have been, we barely got through the two selections we had to work with, and this took nearly 6 hours to complete. We were lucky to have some time for additional experimentation after we completed the blind comparisons.
As to fatigue, I already posted that we took plenty of breaks since careful level matching needed to be done prior to evaluating each pair of CD players.
Having said all that, the participants agreed upon the two selections ahead of time. There were number of reasons for going this route. We could have chosen any acoustic performance, but we selected classical because it covers a broader spectrum of instruments and voices than say a guitar/vocal piece. It could also have been a highly regarded album that is the result of an outstanding recording studio project, however only a couple of participants have ever been in a studio and only one of us has ever experienced a recording session (in both control and recording rooms). We could have also included a live jazz, folk or rock album and there are many good ones. However, most of these are recordings of amplified instruments and voices through PA systems
just way too many variables to account for.
Further, our concern for the tricks (mixing, multi mic, compression, reverb, etc.) applied to most studio and live projects often result in an unnatural sonic presentation making it tough to evaluate what sounds real. Dont get me wrong, there are studio recorded albums that are outstanding and thoroughly enjoyable. It would be interesting to see how the results compare when evaluating these players with such recordings. However, even the very best of them would not be my first choice if I had limited time to compare CD players of any gear.
If an audio component or system reproduces a well recorded acoustic music in the most impressive and believable manner, then I know I am going in the right direction. HERE IS THE MOST IMPORTANT PART: You will only know if the component or system is performing well if you go to enough live performances so you can relate these experiences to what you are hearing from your gear. In this way you can more accurately evaluate the quality of the recording, the naturalness of the instruments & voices and nuances (harmonics, textures, dynamics, micro and macro passages, phrasing, etc.). Furthermore, when well recorded acoustic music is done right, I can accept that I can better evaluate the CDs from studio recording efforts (not the other way around). In this manner, it is the live event or how you experience it that becomes the reference, not the recording and it is the combination of keeping this live event perspective in mind along with the well-selected recordings used that will reveal the less precise, more analytical, and more generally, the inferior players.
Too often audio gear is selected based on which ones make our favorite recordings sound good, but the listener rarely goes out to a live concert. Despite owning different systems and having varied musical tastes, all of the participants have been to multiple concert and symphony halls and cathedrals. The orchestral and choral (with small instrumental ensemble and organ) pieces selected served us well as the means for the group to evaluate the performance of these players.
Kind regards,