i guess in my own experience, there is a fairly decent correlation between large woofers such as the single 12 inchers in big harbeths and spendors with the need for amplifier damping factor and control of the driver and its significant motor structure@jjss49
What's been happening there is the impedance of the woofers in your experiences has been the real issue. To give you an idea of what I mean, my speakers at home employ dual 15" woofers made by TAD. The speakers are fairly efficient; about 98dB and together are 16 Ohms. Our MP-1 preamp has a fairly gutsy direct coupled output and can drive the speakers to a conversational level and the bass sounds just fine. This is a tube circuit that makes about a watt. The damping factor is non-existent- the preamp is balanced and designed to drive 600 Ohms, not 16, yet the bass is fine. The line stage performs well as a headphone power amp; most 'phones are 32 Ohms so 16 isn't that much of a stretch.
I don't think this would be the case if the woofer array were 4 Ohms. That would have an enormous effect! Not so much due to damping, but simply because the distortion of the tubes would go up, causing it to sound fat and possibly muddy.
Its distortion that you want to keep in check. A high damping factor is not needed on almost any speaker- the highest any speaker might need might be 20:1- certainly no more! Damping by itself is really over-rated. In the case of the Harbeth 40, you have a single 8 Ohm woofer in a large-ish box that is ported. This is a benign load for a tube amp but to get the most out of the woofer if using tubes you'd want the amp to have good bass extension at least an octave or two below 20Hz so as to minimize phase shift.