I totally agree; bad recordings can certainly diminish the pleasure of good performances. On the other hand, good recordings don't improve bad performance, they just make the bad performance more obvious. But oh my, when you couple excellent performance with a high quality recording, played on a nice system, you have it all. Maybe the payoff is worth the trouble.
Some "music lovers" will be critical of this view. Some seem to be able to appreciate the underlying music, regardless of the quality of the reproduction.