There are quite a few factors in each system, starting with maybe power to the home, room resonance problems, equipment not meant to play well with certain other equipment, and yes even yourself.
In my system/room setup, I can hear the system distort just before the room is saturated. This happens at way above my normal listening levels. Sometimes my ears say 'no way' before the music does.
Funny thing though, there are even times when distortion is present and I don't mind, but that is an exception for sure.
To make a guess in your system, I'm thinking speakers and room placement. Maybe even some first reflection treatment. But yah, speakers.