Absolute power is only one of the dimensions that matter: obviously, if you are trying to drive a difficult speaker load at say 80db efficiency, a lot of power is required. Conversely if you are driving 110db horn speakers a humble 2A3 valve amp pushing out 2-3 W is plenty sufficient. Generally bigger power means bigger power transformers and consequently greater emission of EMI, the shielding in other words gets more challenging.
Where it really gets interesting: a clean renderer and DAC, i.e. units with a very low noise floor can increase the perceived loudness. Since the amp has to amplify only clean signal rather than a lot of grunge on top, the perceived result appears louder. NB: there is no increase in power, just perception.