I notice at lower volumes a lot of the imaging and clarity disappears.That's a problem with the Yamaha. Going to higher power won't help it.
There is this thing called the 'first watt', which is the idea that the first watt of power is the most important. Apparently the Yamaha isn't doing that bit fairly well. I suspect if you look at its specs, at lower power levels the distortion might be quite a bit higher than at higher levels.
Better quality amplification is better at that problem. If you go with tube gear, even better...