There is no damage that will happen in using an output transformer into an unmatched load, however the amount of power available will be greatest into a matched load. Ie 8 ohm tap, 8 ohm speaker.
That said, the damping factor of the 4 ohm taps should be almost double that of the 8 ohm taps. That may be the source of your reduced dynamics issue; overdamping of the woofer. The damping factor (output impedance) will interact with the design of the woofer differently, and since there is no clear guidance, as others have suggested, it seems best to pick the sound that you prefer.