Do distortion 's affect enjoyment of speaker?

I hope not, since all speakers produce it in one form or another.
Seriously though, from my own experience it can if it a type to which you are sensitive. When you are aware of a certain bothersome distortion or sonic anomaly of any type, it consciously or subconsciously gets in the way of your enjoyment of the music.

Complicated topic. There's no doubt that tube amplification introduces significant distortion, yet many people prefer it to solid state. When comparing solid state designs, its never been proven that distortion relates in any meaningful way to musicality or enjoyment. However, low distortion has become perhaps the major selling point for many manufacturers of high priced solid state equipment. But the way different manufacturers define and measure distortion will probably never be standardized.
Bottom line: Subjectivity reigns supreme.

It depends.

All speakers produce distortion.

But it varies what people are sensitive to, and indeed there are some studies that show certain distortions are preferred.

Thanks
Bill

Distortion doesn't just mean noise or hum or something objectionable. It simply means the speaker's output differs from the input. As stated above, many people prefer speakers that add bass, amplifiers that round off the top end or depress the midrange. Plus, any speaker's output in a real living space is likely going to be far off from what the manufacturer intended as rooms can reinforce or cancel bass, reflect or absorb highs, etc... That's why the only way to judge a speaker is to install it in your system and see if you like it.