I think it's true for most of us as well. However, a consideration - especially for a tube amp, where watts is not quite so cheap - is how badly do you need to avoid clipping?
If an amp clips very gracefully (i.e., it only gently compresses the loudest dynamics and sounds good in the process), then it might not make financial sense to buy an amp powerful enough to never clip.
My solid state amps are all large enough to easily stay in the linear range, but I do have a couple small tube amps that I probably drive into clipping on the very loudest peaks. Nevertheless, the nature of these amps is apparent even at very low volumes, so I don't think their clipping behavior explains the "tube sound".
If we were discussing guitar amps, the story would be very different, since distortion is intentional there.