The biggest danger from what you suggest is that sooner or later you (or a family member) will accidentially turn on both amps at the same time. BIG BOOM! You will loose at a minimum a fuse in one amp, you might loose 1 amp, you might loose 2 amps, you might loose a couple of drivers in your speakers. You are also adding a load which may or may not have a noticible effect of the sound of the speaker. DON"T DO IT. :-)
There was a thread in Audio Asylum a month or two ago on this exact subject - one of the posters identified a switching devise (I forget the name of the manufacturer) and I looked up the web site. It looked like it could work quite well (on paper).
Personally I'd continue to use the banana plugs, no accidents or sonic degredation to worry about.