The main concern with a speaker with a low nominal impedance, such as 4 ohm, and tube amps has to do with frequency response changes due to the relatively high impedance of the amp interacting with the impedance of the speaker. While 4 ohms might be the nominal impedance, the actual resistance at each frequency is probably quite different, which means certain frequencies will be emphasized more than other depending on the speaker's actual resistance at that particular frequency. If a 4 ohm rated speaker had a perfectly flat impedance curve (resistance at all frequencies the same) the impedance match of the speaker and the amp would not matter. But, all speakers have a pretty wide ranging up and down curve regardless of their single number "nominal" rating. Hence, all 4 ohm speakers are not the same when it comes to how they will react to the impedance of the amp as far as frequency response change.
Ideally, the amplifier's output impedance will be so low compared to the speaker impedance that there will be not much in the way of frequency response effects. A tube amp with a 4 ohm tap will have enough enough of a turn ratio in the output transformer so as to lower the output impedance to make it work reasonably well with a 4 ohm speaker. But, there is no free lunch, so this means a sacrifice of some power that can be delivered to the speaker. If the amp only has an 8 ohm tap, it may not have a low enough output impedance to avoid major effects on the frequency response when working with a 4 ohm speaker. Whether or not this effect is severe enough to disqualify the amplifier depends on the particular impedance curve of the speaker, the actual output impedance of the amp and the particular taste of the listener. I've heard a number of supposed mis-matched speaker and amps that sounded quite nice.
