If you leave both amps connected to the speaker, the one powered up will likely damage the output stage of the one powered off. That's why you need a switch.
Also, if either amp uses tubes in the output stage be sure that the speaker is connected to the amp before powering up or you could damage the output transformer. No such problem with solid state outputs.
Also, if either amp uses tubes in the output stage be sure that the speaker is connected to the amp before powering up or you could damage the output transformer. No such problem with solid state outputs.