IOW, it can get a whole lot better, but to do so you either need a lot more power (not available from an SET) or get a speaker that is a good 10-12 db more efficient! This despite how good it seems to sound right now; but the OP is correct in his surmise that more power (or more efficiency) will help things out.
The problem is that SETs have bandwidth limits that get more pronounced with the greater power they can make. That is why the lower power units sound better.
BTW, there was one comment about push-pull having crossover distortion; this is not true in most cases. Crossover distortion is an artifact of class B operation, and to my knowledge no home audio gear employs class B, so that's a bit of a red herring. It is the most common complaint against push-pull, but as best I can make out, is mostly borne out of not understanding push-pull technology all that well.