When have A/B comparisons led you astray?

It's almost impossible to AB anything without bias, for a variety of reasons...
1. When presented with 2 options, the second is almost always preferred because we naturally tend to evaluate the second relative to the first (but not the other way around). Since the second is usually the 'new' one, we are biased to the new.
2. Warm up time. As most know it takes 30 minutes or so for something to warm up. The interim delay is too long to retain realistic recall.
3. Visuals really matter. We are strongly biased to something that 'looks better' whether we realize it or not.
4. We are biased to the new and/or more expensive, again whether we realize it or not.
5. It's impossible to get a good sense of things in a short time. Long listening sessions are needed. Again, by the time you've listened to the second for a while, you've completely forgotten what the first sounded like.

Not sure it's possible to truly compare in AB testing. Of course huge differences in sound are readily apparent, but subtle ones much less so. Lesson - don't make changes unless they are huge ones. Small 'tweaks' are purely emotional.

What I find interesting is that fact that some changes that I've made (DIY speaker cables for example) seem to have resulted in a positive result the majority of the time, but a few song really don't sound right anymore. How does one choose what music to use for the comparison because each recording had different characteristics. When I added the Nordost Sort Kones I only noticed a positive result across the board, but maybe that placebo talking.

Placebo effect. My brother went to audio store to buy Cabasse speakers. Salesmen presented first speaker that sounded very good. Then he played a little more expensive speaker that sounded much better for a little more and finally even more expensive speaker that sounded incredible but my brother could not afford it. He then left my brother in silence for a while taking care of another customer and came back with another option - brand new Cabasse design that plays almost as good as the latest most expensive one, but cost much less. After listening - my brother said that this is a great bargain, sounding almost as good as the most expensive one, an bought it. At the end salesman told him that it was the same, least expensive speaker he listen to first.

"i just change stuff. period, and don't worry about it"

- So do I. I found that switching A/B is not as good as playing one component for a week with different types of music then do the same with the other. Constant switching between components to see what improved and what not has less value to me than general impression after week of listening.