Does the power of an amplifier really matter?

It depends greatly upon the design of the amplifier and the characteristics of the speakers that are used with it.

In many cases, it is only the first watt or two which make the most impact on the sound. However, as playing gets louder,particularly with some inefficient speakers(and some with difficult impedances) the amplifier can become strained, and can eventually go beyond its ability to drive the speaker cleanly.

Current delivery ability of the amp can become increasingly important in these situations, and also when deep bass response in in the equation.

Since it requires about double the amp power applied, to increase most speakers SPL by 3db/1 meter, it is apparent that higher amp power can be useful to extend the SPL of many speakers up to their maximum SPL capability.

Additionally, dynamic headroom may play a factor in this, and many amp makers will include a dynamic headroom spec, which means that their power supplies are robust enough to provide "bursts" of power beyond their rated power levels, to handle difficult musical passages.

The more modest your listening SPL demands are, generally the less demanding you are of amp power, unless your speakers present a particularly difficult load for the amplifier to handle.

I realize that many other technical factors are present, but I'm trying to explain it in simple layman's terms.

"So, if I listen to my stereo at a volume of say, 70 dB, what is the difference in a 100W amp vs. 250 W amp? If I listen at 70 dB, is more power delivered to the speakers with a 250W amp vs. 100W amp (resulting in better driver control)?"

No. A speaker needs a certain amount of power to play a particular note or tone at 70 dB (in your room, at your listening location). That's exactly what the amp is delivering at that moment, whether it's a 100w or a 250w amp.

The advantage of a 250w amp is that music, unlike a test tone, isn't at constant loudness. Occasional peaks can really drive up a speaker's need for power. With a more powerful amp, you're less likely to hit its limits or start to hear noticeable levels of distortion when the cannon goes off in the 1812 Overture, say.

"Is it that a higher powered amp delivers a given amount of power more cleanly?"

If you're talking about a large amount of power, that may be the case. Most of the time, however, your speakers aren't demanding very much of your amplifier, and even a small amp will be able to deliver a clean signal.

Keep in mind that an amp's power rating is only a very crude measure of how powerful it is under the range of conditions it would face in a typical music system. Someone who knows what they're doing can do much more meaningful measurements, but the measurements you see on the typical spec sheet aren't particularly meaningful. The best way to tell whether an amp is powerful enough is to bring it home and try it out in your system.

Several personal points here:
1-Yes, every 3db requires a doubling of power. However, it takes +10db to double the perceived volume.
2-70db is a tad too quiet for most people. Most speakers are putting out 1 watt to produce around 89db. Simply to double the perceived volume would require about 10 watts (99db).
3-Finally, keep in mind that although it doesn't seem like a big deal to exceed an amps max rating just for a short period of time, it can be. Some very nice amps handle dynamic headroom really well, while some lessor quality units start distorting rather quickly and don't recover for a while. To put that into perspective, compare a 7 watt cary amp to a 100 watt klh receiver. Both playing at 5 watts will sound drastically different. In light of what Pabelson stated above, the 1812 overture may not be as loud on the Cary, but it sure will sound better and handle the peaks with no noticable problems.