Hazyj, There is something to the idea that smaller amps sound better. In the case of push-pull tube amps, this has almost entirely to do with the output transformer itself. The reason for this is that the more power the output transformer can handle, the less bandwidth it has. So in the larger designs, usually the designer has made a tradeoff based entirely on what he thinks is important.
If the amp employs negative feedback, usually the high frequency bandwidth is the area that is sacrificed. This is because there is not a lot of energy at high frequencies so its possible to get away with this if you also run negative feedback to help output a little with the voltage response of the amp which will be degraded (another way of looking at this is the output impedance will be higher at the area of the rolloff and beyond).
In the old days, 60 watts was about the limit where the output transformer could make the bandwidth required to really be hifi. That bandwidth BTW is well past 20Hz-20KHz; its nice to get response up to 100KHz if you can so phase shift at 10KHz is not adversely affected. This really helps with the presentation of the soundstage, depth in particular.
These days designers seem to have pushed that power level up to about 100 watts before bandwidth really starts to suffer.
Now the problem with push-pull is often that there is a phase-splitter circuit that introduces some distortion. This is not true of all P-P amps, but it is true of most of them. This generally is not an issue until you get into the lower power regions of the amplifier, at which point the distortion of the phase splitter comes into play. For this reason a highly efficient speaker (+98 db or so) may not be the best choice with a plus-100 watt amplifier as you may never get the amp out of that region of higher distortion at lower power levels.
SET amplifiers get around this problem by not having a phase splitter (that's how we do it too although our amps are push-pull). This allows them to have a distortion characteristic that becomes unmeasurable as the power decreases to zero. To really take advantage of this, you really do need a high efficiency loudspeaker! The reason is two-fold with SETs- first, that bandwidth limit discussed with the OPT of P-P amps is more profound with SETs, the practical upper limit being only about 7 watts before bandwidth is degraded (for this reason many SET designers don't bother trying to get the bass right in the OPT as the speakers that have that kind of efficiency don't play the bottom octave either). The second reason is that you never want to push an SET past about 20% of full power or else the higher ordered distortion products come into play (when this happens, its usually on transients, and because the human ear uses the higher orders as loudness cues, the result is that the SET sounds a lot more dynamic than it has any right to because the loudness cues are occurring on the transients. You read about this 'dynamic' character with SETs all the time, but its an indication that the speaker used is not efficient enough).
Where is is all going is that if you want to really get the most out of your tube amplifier investment dollar, the selection of the loudspeaker is really important!
Something else to be aware of: Tube amps make more power and run more efficiently into higher impedances. You really don't want to make **any** amplifier (tube or transistor) work all that hard- the harder you make it work, the more distortion it will have. So you want to avoid 4 ohm speakers generally speaking- the amp will make less distortion into 8 or 16 ohms. This has nothing to do with whether the amp employs negative feedback or not- and it has everything to do with the fact that OPTs operate more effectively (wider bandwidth, lower distortion, less amplifier power converted to heat) into higher impedances. This is why you are seeing more 16 ohm speakers on the market these days- designers are slowly figuring this out.