A lot of comments here that fit my experience. I own, and like, SET amps. Yes, there are things that they do that really cannot be matched by other designs (realistic decay of notes, truly enveloping soundstage).
But, the difference between the SET topology and pushpull is perhaps not as big a difference as that between tube types. I agree with the poster who noted how much a 300b in SET and pushpull share common characteristics. I have heard quite a few 300b, 45 and 2a3 pushpull and SET amps. I own, and like the sound of my pushpull 45 amp, even though it doesn't do all that a good SET can do. The amps tend to sound most like the tube type, with pushpull topology sounding "tighter" and seeming to not have as much "bloom" and not delivering all of the harmonic richness of the SET.
On the power limitation/speaker compatibility issue, I would agree that, if one happens to like a speaker that demands more power, then the priority is to find a compatible amp, and if that means foregoing certain topologies, that is just one of MANY compromises one must make when building a system (to me the priority is speaker choice first and foremost, then finding a compatible amp--not the other way around). But, I find that most people overestimate how much power they really need and the range of speaker choices for SET amplification is really not as narrow as some would believe. I've heard several 845 and 211 SETs that would work with almost anything, provided the room is not too large and one does not require extremely high playback levels. If one finds that much more power is needed, I would look at OTL amps first, then solid state amps (I tend to dislike higher powered pentode tube amps).
On the subject of OTLs, it may well be the case that a good OTL would be a first choice, regardless of power requirement. I love the speed, liveliness and "directness" of OTL. Within their power limitations, SETs can also have that quality, but, it is really hard to beat an OTL in those regards. To me, the tradeoff is that the sound is a touch "raw" or "rough."
One more thing, there is a pretty wide range of capabilities of SET amps. Although the SET topology is quite simple, SET amps are actually quite costly to do correctly because of the demand placed on the output transformer. In order to handle the DC current in the primary, the transformer must be big, air gapped, etc. SET transformer design requires even more complicated balancing of tradeoffs than pushpull transformer design. The better SET amps tend to be quite expensive.