When have A/B comparisons led you astray?

I find that A/B comparisons require time, especially when performed when in one's own system.

Sometimes, just the fact that something sounds different can lead to one feeling that it either sounds better or worse, initially.

It is only over time that we can determine that the difference is really better or worse, provides more of what we want or less of what we don't want (or vice versa). Many people A/B in stores in short listening sessions. To me, the problem with this can be multi-fold, but typically evolves around an evaluation that is based on too narrow of a field of characteristics.

Also, I find that if one feels they have a deficiency in their system, they tend to over-correct for that deficiency and sometimes at the cost of other areas of importance.

So A/Bing is necessary, but is part of a longer term process. The good news is that Agon allows for longer term A/B testing by smartly buying used with the ability to sell at zero to little loss.

FYI - I am in this process currently. Having recently brought in an Esoteric and Levinson integrated CD players (with Dacs). I will also be bringing in an additional amp and preamp. Giving me at least two or more of each component and hopefully allowing me to find the best synergy in a final system. I know this will take some time, but I also know that I can do this and then sell what I don't select at a reasonable end cost to myself.

Not being able to A/B components at home ends up being more costly and keeping one on the merry go round for too long.

It's almost impossible to AB anything without bias, for a variety of reasons...
1. When presented with 2 options, the second is almost always preferred because we naturally tend to evaluate the second relative to the first (but not the other way around). Since the second is usually the 'new' one, we are biased to the new.
2. Warm up time. As most know it takes 30 minutes or so for something to warm up. The interim delay is too long to retain realistic recall.
3. Visuals really matter. We are strongly biased to something that 'looks better' whether we realize it or not.
4. We are biased to the new and/or more expensive, again whether we realize it or not.
5. It's impossible to get a good sense of things in a short time. Long listening sessions are needed. Again, by the time you've listened to the second for a while, you've completely forgotten what the first sounded like.

Not sure it's possible to truly compare in AB testing. Of course huge differences in sound are readily apparent, but subtle ones much less so. Lesson - don't make changes unless they are huge ones. Small 'tweaks' are purely emotional.

What I find interesting is that fact that some changes that I've made (DIY speaker cables for example) seem to have resulted in a positive result the majority of the time, but a few song really don't sound right anymore. How does one choose what music to use for the comparison because each recording had different characteristics. When I added the Nordost Sort Kones I only noticed a positive result across the board, but maybe that placebo talking.

Placebo effect. My brother went to audio store to buy Cabasse speakers. Salesmen presented first speaker that sounded very good. Then he played a little more expensive speaker that sounded much better for a little more and finally even more expensive speaker that sounded incredible but my brother could not afford it. He then left my brother in silence for a while taking care of another customer and came back with another option - brand new Cabasse design that plays almost as good as the latest most expensive one, but cost much less. After listening - my brother said that this is a great bargain, sounding almost as good as the most expensive one, an bought it. At the end salesman told him that it was the same, least expensive speaker he listen to first.