Do distortion 's affect enjoyment of speaker?

I hope not, since all speakers produce it in one form or another.
Seriously though, from my own experience it can if it a type to which you are sensitive. When you are aware of a certain bothersome distortion or sonic anomaly of any type, it consciously or subconsciously gets in the way of your enjoyment of the music.

Complicated topic. There's no doubt that tube amplification introduces significant distortion, yet many people prefer it to solid state. When comparing solid state designs, its never been proven that distortion relates in any meaningful way to musicality or enjoyment. However, low distortion has become perhaps the major selling point for many manufacturers of high priced solid state equipment. But the way different manufacturers define and measure distortion will probably never be standardized.
Bottom line: Subjectivity reigns supreme.

It depends.

All speakers produce distortion.

But it varies what people are sensitive to, and indeed there are some studies that show certain distortions are preferred.

Thanks
Bill