Damping factor plays a role in system performance, just not a large one.
Damping factors above about 15 or 20 to 1 will result in no audible improvement, but such amplifiers that do have such damping factors have proven themselves incapable of playing definition and body in bass instruments, and for that matter punch is a problem too. IOW things get worse.
This is because such amplifiers employ large amounts of feedback to this effect. Feedback, while lowering output impedance, simultaneously restricts dynamic peaks and impact. Additionally as Duke mentioned above, odd-ordered harmonics that the ear uses as loudness cues are actually enhanced, causing an amp with feedback to be harsher.
The solution is less feedback.
We do perceive a different phenomena as the output impedance of the power amplifier is raised, and approaches that of the speaker- the amplifier will make more distortion.
We have words for this type of distortion (in order of increasing distortion): warmth, bloom, fatness, turgid and finally, muddy.
So the trick is to have an amplifier that ideally has at least 4:1 over the speaker in use. For a 4 ohm speaker this puts the amplifier output impedance at about 1 ohm, 0.5 ohms is likely to be near ideal.
Note that the output power of the amplifier is what places the diaphragm of the speaker at any position in space that it is capable of. The idea that the speaker diaphragm 'falls' back to zero and can ring really does not happen- the speaker diaphragm is powered to full excursion and then powered back to zero- power is on the driver at all times.
Due to this reality, damping plays a minor role.