When have A/B comparisons led you astray?

Same mechanism. Neural adaptation and it is a common source of bias in long term even more than with A/B comparisons.

I recall a study done at a hifi show. Subjects made a blind preference choice between two options (speaker vs. speaker, or tube amp vs. SS) in a level-balanced setup. In every two-way test, the subjects preferred the second choice, regardless of what it was.

Kal

I find that A/B comparisons require time, especially when performed when in one's own system.

Sometimes, just the fact that something sounds different can lead to one feeling that it either sounds better or worse, initially.

It is only over time that we can determine that the difference is really better or worse, provides more of what we want or less of what we don't want (or vice versa). Many people A/B in stores in short listening sessions. To me, the problem with this can be multi-fold, but typically evolves around an evaluation that is based on too narrow of a field of characteristics.

Also, I find that if one feels they have a deficiency in their system, they tend to over-correct for that deficiency and sometimes at the cost of other areas of importance.

So A/Bing is necessary, but is part of a longer term process. The good news is that Agon allows for longer term A/B testing by smartly buying used with the ability to sell at zero to little loss.

FYI - I am in this process currently. Having recently brought in an Esoteric and Levinson integrated CD players (with Dacs). I will also be bringing in an additional amp and preamp. Giving me at least two or more of each component and hopefully allowing me to find the best synergy in a final system. I know this will take some time, but I also know that I can do this and then sell what I don't select at a reasonable end cost to myself.

Not being able to A/B components at home ends up being more costly and keeping one on the merry go round for too long.

It's almost impossible to AB anything without bias, for a variety of reasons...
1. When presented with 2 options, the second is almost always preferred because we naturally tend to evaluate the second relative to the first (but not the other way around). Since the second is usually the 'new' one, we are biased to the new.
2. Warm up time. As most know it takes 30 minutes or so for something to warm up. The interim delay is too long to retain realistic recall.
3. Visuals really matter. We are strongly biased to something that 'looks better' whether we realize it or not.
4. We are biased to the new and/or more expensive, again whether we realize it or not.
5. It's impossible to get a good sense of things in a short time. Long listening sessions are needed. Again, by the time you've listened to the second for a while, you've completely forgotten what the first sounded like.

Not sure it's possible to truly compare in AB testing. Of course huge differences in sound are readily apparent, but subtle ones much less so. Lesson - don't make changes unless they are huge ones. Small 'tweaks' are purely emotional.