When does speaker distortion become audible?

There exists an astonishing amount of differences in capability of speakers to render the signal accurately/beautifully. It is common when comparing speakers to have the sense that one is far more clean/clear/distortion-free than another.

The other half of what Douglas said is that our memory is very poor & gets there very quickly. We remember in a "somewhat" fashion. Lots of speakers may sound great until we do side by side auditions.
Overall, it was good or not quite good. And lots of us have that problem. It's never as accurate as gettig a "bad ice cube."

It becomes audible when you can hear it.
Next question......

Gvsale, not only is there the issue of our memory over time, but if we are not comparing speakers in the same setup/system then the entire "apples to apples" assumption is lost. Then one can only make a generalized statement regarding the comparison. But still, I assert that the differences are so fundamental, so great that one can obtain shall we say a tentative conclusion in regard to performance of differing speakers until such time one might be able to compare directly, if ever.

Imo audiophiles who have not heard a great number of speakers, i.e. dozens to hundreds, have less understanding of this gulf in performance which exists in speakers. As Gvasale says, when moving from one to another rig in a different location it is not as apparent as when hearing several speakers in one setting. Shows are great for revealing such differences in performance, even though on different systems.

I am still amazed after 30 years in audio at how much variance is possible between different brands and technologies of speakers. It seems there is a speaker (house sound/technology) for nearly every ear out there! (It's just a matter of finding it!) :)

Usually when a speaker is overdriven.

Speakers may also be more victims than cause of most other forms of noticeable distortion that may also be way more common and also harder to diagnose and correct when present.