"Damped" and damping factor are of course not the same thing. Damping factor is a simple output impedance / 8 Ohm calc. "Damped" refers to how or the characteristic of how a system will respond to a step response.
Speakers are designed to be driven typically by constant voltage sources. Dynamic drivers on the other hand can behave better, i.e. lower distortion with constant current sources. To that end, all speakers are the product of trade-offs, and designing for 0 output impedance forces other trade offs a particular designer may not want to make. The amplifier is part of the circuit that governs how the speaker will behave. We can talk about a "perfect" world, but dynamic drivers are already imperfect. So better to discuss which imperfections you want to live with.