Anyone concerned about not being able to compare or judge because they will forget what they heard in a few minutes should stop and think about that. Ask yourself, if you really can't remember, then why would you care?That isn't the issue, the problem is you have no music in the hour(s) that wait for the tube amp to cool down and warm up.
A high quantity speaker A/B switcher is the solution, but you have to make sure the tube amp output is connected to a load at all time while power on.
A safe way to swap is mute the input, if the switcher is on speaker A, select A+B first then change to speaker B, so the tube amp will be loaded while swapping speakers.