It really depends on one's speakers and system setup on how much power you need... With mine, they demand 0,01-0,01W with 0,2W peaks when I listen to them loud. So, my needs are below 1/5 of a watt... 1 watt for me is already splurging, and I rarely go there, only for concert-level experiences.
In general, as the amplifier gets more powerful, it can only be done through additional gain stages in the amplifier that erode the soundfloor. (Which is the low level details).
Most speakers are completely incapable of reproducing low soundfloors, so there's seemingly not much left missing as one climbs to even 5000WPC with the so pervasive "I need an arc-welder just to nudge"-type loudspeakers. Yet, provided sensitive speakers, going from 1W to 20W can already be a deal-breaker as already a noticeable layer of the soundfloor is lost, which might be subtle when you hear it first time in your life - yet, when you live with it, and are used to it, it will mean a LOT to loose it....