A 3 dB increase in sound-pressure level requires a doubling of the amplifiers output power. A four ohm speaker requires more power to drive than an eight ohm speaker.
A loudspeaker with a sensitivity of 88dB at 1W would produce 91dB with 2W, 94dB with 4W, and 97dB with 8W, for a speaker with an 88dB sensitivity to produce a 109dB peak would require a 128W amp. An speaker rated at 91dB would only need 64W to reproduce the same peak.
That means an 8 ohm speaker will be easier for you amp to drive. The ability for your amp to achieve those peaks, which occur in virtual all music is greater when you use the more efficient speaker, so 8 ohm loads are easier to drive.
So your last sentence is correct, it will take twice as much power to drive the 4 ohm load in such a way as to reproduce all the action of the music.
This is based on the information that Robert Harley wrote in his book "The Complete Guide to High-End Audio."