This is not an easy question to answer, and it would depend on particulars that are impossible to factor in, such as your own personal preferences and priorities. My own priority is for the amp to sound good while playing at lower output levels where the vast majority of my listening takes place. For me personally, I happen to favor the sound of low-powered triode tubes, and even in the solid state realm, the amps I like tend to be lower-powered, such as the gear from First Watt. This means using fairly efficient speakers (mine are 99 db/w efficient).
Millercarbon raised a very point--lower efficiency speakers cannot really achieve good dynamics no matter how much power is fed because of thermal compression. As more power is fed to the voice coil, it heats up and that heating increases resistance which reduces the amount of power that can be delivered to the driver. The less efficient the speaker, the greater the compression. My difference with his opinion is that I would not be so dismissive of the Harbeths just because they are not efficient; there are plenty of other reasons to like any particular speaker even if it cannot do "really loud."
It might be the case that the higher powered Pass amp will improve the sound, even at lower levels of play, but, I have my doubts about this because caused by some inherent superiority of having much more power in reserve. This is something you will have to try for yourself to see if it is the case. A long time ago, I auditioned an am rated to deliver something like 50 watts/channel. The same company made a 200 watt/channel amp that the dealer said was very similar in design and parts quality, but used more output devices to achieve the higher power capability. I actually preferred the sound of the lower powered amp in a direct comparison.
