I wouldn't do it. You will get considerable drops in impedance that could be catastrophic. You can get an impedance matching transformer that will allow for multiple pairs of speakers without jeopardizing the amp.
My opinion-the speakers don't sound better at 4 ohms, you like the sound of the amp better when it is running at 4 ohms.
My opinion-the speakers don't sound better at 4 ohms, you like the sound of the amp better when it is running at 4 ohms.