Depending on the concerning ear, everything you put into the signal path will degrade the sound.
Years ago, I needed to do what you want. But I needed to be able to switch 4 turntables, 6 amps/receivers, 2 aux and 12 pairs of speakers. Plus, spares of each. I had a hard time finding the switches i needed. And wiring them together would be a rats nest pain. That's why you pay the big bucks.
So, I made a Patch Board. Pamona banana jacks for power outs and speakers. RCAs for turntable, aux, pre out and amp in. Using patch cords I could switch. ab, mix every which way. Was not a quick turn or push of a switch but did the job. And all the connections were right in front of me.