If you load the 8 ohm tap with a 4 ohm load, the plate to plate impedance that the tubes are supposed to be driving will be reduced. This will result in less power and increased distortion (the 'loosness' that others have mentioned).
This is harder on the tubes, as the loss in power is dissipated in the tubes themselves.
The transformer will also be over-damped, resulting in a loss of high frequency bandwidth.
Ultimately, if you are investing in a tube amplifier (*any* tube amplifier), your tube amplifier investment dollar will be best served by a speaker that is at least 8 ohms as opposed to 4, all other things being equal.
If you can go with 16 ohms things get even better, but some amps these days don't have a 16 ohm tap, which is really too bad.
The interface between the amp and speaker is paramount to getting the performance out of any amplifier, transistors included.