I'm going to use a couple of car analogies.
If I am taking a scenic drive it is much more enjoyable in a clean Lexus than a rusty Yugo with dirty windows. It's the same scenery but the experience of driving a nice car is, for me, much better. I can enjoy the drive in both cars but give me the Lexus any day.
Decades ago, in the age of AM car radios, a guy I met at a conference turned out to be an audiophile and we were discussing our hobby. I mentioned the "goose bump test" for audio systems and he said, "Yeah, but I can get goose bumps listening to the car radio." I thought for a second and agreed.
Bottom line, I like listening to music whether it's high fidelity or not. The only exception is a few rock concerts I've attended where the sound was painfully loud and distorted. But give me good music with a poor recording and I can get plenty of goose bumps.