First, I wouldn’t say that 4 ohm speakers are **necessarily** harder to drive than 8 ohm speakers, as there are many other variables involved. Including the efficiencies of the speakers; how the magnitudes of their impedances (the number of ohms) vary over the frequency range; the phase angles of their impedances at various frequencies (which describe the degree to which the impedance becomes partially inductive or partially capacitive at various frequencies, rather than purely resistive); etc.
But yes, typically a speaker having a low nominal impedance such as 4 ohms will be more difficult to drive than one having a higher nominal impedance such as 8 ohms. Adding to what has already been said, perhaps a good way to envision that is to consider a pair of extreme examples.
On the one hand let’s say that all the amp is driving is the input impedance of another amplifier, as would be the case if the amp were only connected to the speaker-level input of a powered sub. It might then be seeing a load of perhaps 100,000 ohms, which would result in the sub responding to the voltage being put out by the amp in question at any instant of time, but per Ohm’s Law (thanks for bringing that into the discussion, Ghosthouse) drawing essentially negligible current from that amp. In that situation the amp in question would hardly know that it is connected to anything at all, and the power it would be putting out would be essentially zero. (Power into a resistive load equals voltage x current). (In saying this, btw, I'm putting aside the fact that tube amps having output transformers should not be operated unloaded while processing a signal, that being a separate issue).
At the other extreme let’s apply a load of essentially zero ohms to the amp, by shorting its + and - output terminals directly together with a heavy gauge jumper. I think most will recognize that the amp would be incapable of putting any kind of reasonable signal into that near zero ohm load, because per Ohm’s Law creating a non-zero voltage across a zero ohm resistance requires infinite current. And as the amp attempts to do that the result is likely to be either that it goes into a self-protective shutdown, or a blown fuse, or damage.
Obviously a 4 ohm load comes closer to being a direct short than an 8 ohm load, and an 8 ohm load comes closer to being a negligible load than a 4 ohm load, so there you go!
One additional point: As Ralph/Atmasphere has stated here many times, for various reasons both solid state and tube amplifiers will exhibit measurably better distortion characteristics when driving 8 ohms than when driving 4 ohms.
Regards,
-- Al