... how important is Damping factor in determining what amp to buy

When a signal is sent to a speaker to play a note, the cone vibrates. When the signal stops, the cone is still moving back and forth. and is an electric motor. This movement sends the electrical signal back to the amp, and a highly damped amp will stop the cone sooner.

Amp won't make much difference in damping, since there are already many ohms of the speaker's voice coil in series.  Overall effective damping factor is around df=1.5 at best.  

Three plus years ago I was "on a mission" to find a new pair of floor-standing speakers.

One of my stops was at a dealer to audition the then Magico S5's.  As the "main" listening room was occupied, the salesman hooked the speakers to a Hegel power amp with a very high damping factor.  When the "main" listening room became available, he wheeled the S5's over there and connected it to a pair of large tube monoblocks, with a very low damping factor, and driven by tubes, the Magico's sounded much more "musical" and had more bass.  I suspect that, the pairing of a low damping factor and the Magico bass driver just worked better together.