Thanks @holmz for the various very good points and suggestions. Just a clarification on one point, as this is one of the things I'm trying to understand here:
I doubt that the amps are any “faster”, as most amps play 20kHz just fine, and nothing is going to arrive faster from a 2kHz signal, than a 20kHz signal.
With fast I mean that the transients, the typical example being the hit on a snare, are quick and powerful. So in a graph measuring the sound pressure, the peak would be higher and narrower on a "fast" amplifier and lower and flatter on a "slow" amp. The sound is more compressed in a "slow" amplifier and thus the hit of a snare or the initial transient of a piano note will be felt as softer by the ear.
I understand that there are other factors discussed that heavily influence the perception, but I do believe that this is part of the equation. Low powered amplifiers would in my understanding be therefore more suitable to avoid the problems I've had. It is not so much sustained notes that cause issues for me, but the percussive sounds and initial attacks on many instruments.