I commend the OP for starting this thread. And I think his perception that he's barely using a watt is probably accurate. I often advocate the use of a simple app on a smartphone to measure sound pressure levels. I use Decibel X PRO. Using that app I find that I am almost always listening in the 65-70 dB range. I appreciate that there are other, more truly accurate ways of measuring SPLs out there. My approach is "close enough for all practical purposes".
This is a worthwhile thread and might even deserve to become a sticky. It is great to see another forum member referring to Roger Sanders' excellent white paper on tubes vs. solid state. That's useful information, but it doesn't address another topic that I thinks is relevant here.
The topic I'm referring to is circuit design, both within the amplifier and the crossover. Here (IMHO) the circuit design, how it is wired (point-to-point or does it use circuit boards?), parts count, parts quality, the ear of the designer and so forth have a critical impact on the sound of an amplifier. I think that sonic differences are often attributed to power that have much more to do with the topics above, each of which is worthy of a graduate seminar and extensive study.
I used to own an amplifier designed by Roger Sanders. I used it to drive my Quad 63s. It was superb. Do an internet search on "Innersound Electrostatic Amplifier". You might find a reference to a sneak peek by Harry Pearson circa ~2000 in which he hinted at the high praise he was going to bestow upon it. That amplifier had loads of power, no matter how you wanted to measure it.
I am using different amplifiers now. Though I still own Quads (now 2805s and use a Quad 909 amplifier to drive them) most of my listening is through my Altec A7 Magnificents driven by 1.5 watt Alan Eaton 45 monoblocks. IMHO the circuit of the Alan Eaton amps is the quintessence of elegance. The number of parts is tiny. The quality of parts is high. The circuit has been refined over years by the designer. Same with my Supratek preamp.
System synergy is essential when discussing this question and an amplifier that sounds good in one system might sound not-so-good in another system. Listening preferences are also very relevant in this discussion. How do we listen? What do we listen for?
Every piece of equipment is a complex set of compromises. Cost, size, weight, specifications, etc. I have come to a place where I'm mainly interested in equipment produced by small, specialist companies that don't advertise, where the reputation is spread via word of mouth.
Just my .02...