So far I've not seen this mentioned yet, IMO this is one of the most important reasons to have some bandwidth:
Phase shift accompanies bandwidth limitations. In general, you can expect phase shift components to manifest to 1/10th the upper cutoff frequency, so if the amp cuts off (begins its rolloff) at 50KHz we will hear artifacts announcing that at only 5KHz.
This is true on the bottom end as well, phase artifacts will be heard at 10X the cutoff frequency, so a 20KHz rolloff will have artifacts up to 2KHz.
A premature HF rolloff will indeed manifest to our ears as a darkness or slowness in the amp at high frequencies. Oddly, if the rolloff is severe, it can have a brightness and a darkness at the same time! Phase shift can have a pronounced effect on the amp's ability to portray an accurate soundstage as well.
An LF rolloff is heard as a lack of impact. So yes, 2Hz is the minimum cutoff frequency if you want the bass to play right.
The fact that a speaker does not have this sort of bandwidth really does not seem to be relevant- you can hear these artifacts easily despite a lack of bandwidth in the speaker.
Excessive bandwidth on the high end can lead to RF problems. If the amplifier is capable of amplifying broadcast frequencies, you can expect to fry a lot of tweeters. You can also expect the amp to run hot, in the case of transistors. It is for this latter reason that many SS amps are bandwidth limited. It is for the former reason that we limit the bandwidth in our amps (our output section goes well into the megahertz region). Another reason to limit bandwidth is that if the amp is amplifying an out-of-band signal, it can gooble up power that might otherwise be used to drive the speaker with music.
Finally, if you have too much bandwidth you do run the risk of stability. This is particularly true if the amp runs feedback (ours get the bandwidth without feedback, BTW). Due to propagation delay times in the amp, negative feedback can become positive feedback if the frequency gets too high! For this reason feedback loops have to be handled with extreme care by the designer to prevent oscillation (heck, you *always* have to be careful with that anyway), as super HF bandwidth is tricky- you don't want the amp to have a reputation for blowing up, or blowing up speakers!