What contributes most to a change in how an amplifier sounds?

The transformer.

Everyone forgets the power supply. It isn't as sophisticated as the amplifier circuit itself but it's actually the direct electrical connection to the speakers whose impedance interacts with the amplifier to great affect. The thing we think of as the amplifier is basically a valve that opens and closes the power supply supposedly as an analog of the input signal. So the better the amplifier does that the closer the output is to the input albeit larger. But that assumes the power supply is robust and pure. If it isn't even if the amp is accurate it's varying a changing source of current and the result is loss of fidelity.

It turns out many circuits are pretty good but power supplies are expensive and you can cheat with little or no change in the classic measured specs. The only thing that changes with a so so power supply is the sound. Power supply is a major factor why costly amps sound so good. There's money to make pretty good power supplies. But I suspect even 6 figure amps can be improved with even better power supplies.

Distortion signature and output impedance. 

1- whoever is listening.

2- location, local, location.

Power supply 

Power supply

Power supply