is it ok to leave a tube amp on without speakers connected

No, and agree you should never connect and disconnect speakers with the amps on in any case.

Thanks for the responses.  Makes perfect sense.

As previously said, you can burn the output transformers. 
I also have two amplifiers and two sets of speakers. I made a switch box that allows me to choose which amplifier am I listening to, connected to which speakers. There are some models on the market (selectores and switches), but I decided to build my own and I'm very happy with it. 

Depends on certain amps.  I think my previous Cary 211fe didn't need a resistive load connected meaning you could disconnect the speakers.  But I wouldn't bother or mess with it.

Most likely the output tubes will be damaged once you remove the speaker load as the inductance in the primary is converted into a very high voltage and the power tube shorts that voltage to ground. But then again, the transformer may damage as that energy arcs across the windings since it has nowhere to go.