Atmasphere, at the risk of sounding completely ignorant... why hasn't somebody done what you're proposing? This sounds brilliant, simple, and straightforward. In fact, I'd think that this sort of testing would supply much of the sort of information that I'm looking for.
It isn't that I disagree that each piece of equipment won't sound different, and I certainly agree that what sounds 'good' is subjective, but the methodology behind the sound reproduction should be measurable meaning that it should be possible to say something like: okay, you are looking for this sort of sound, for this sort of speaker, etc. Then you should be looking for amps with these characteristics.
The problem as I see it right now is that if none of these measurements have any value, then how can any of these judgments be made. This is why, at least in part, I think your idea is great.
One last note: about not measuring what is important to the human ear. I'm going to go out on a limb and say that this is likely not true. I can't say for sure, but in my field I research a fair amount of psychology regarding military deterrence and I am constantly impressed at how far the psychological and biological sciences have come in recent years. My point is that I would be shocked if no one has studied the interaction of the human brain, our in-built hearing, and how it interacts with music. Any study such as this would lead, inexorably, toward a better understanding of what appeals to the human ear (in general) and what does not. There are cultural issues here besides biological ones, but momentarily putting these aside, I believe you'd find that, in fact, what we hear, how we hear, and why we like certain things can be measured to a certain degree. To extrapolate, this would explain why certain types of sound reproduction--maybe using tubes?--remains popular even though arguably inferior technologically.
