Yeah. Don't.
Nonsense. Or maybe true if you are using a cheap switch. Niles, anyone?
A good selector switch is really a set of time adjusted relays that have little or no effect on the signal pass through them. I own and have used a Luxman AS-50R selector w/ remote for years and cannot hear it in the system. Proper placement and wiring is crucial. I think I paid just under $200 plus shipping from Japan on eBay several years ago.
Luxman makes several models, all good, but it's a lot easier and less disruptive to switch from your listening position. Van Alstine also makes a very good unit (ABX Switch Comparator).
These two companies also make good unbalanced (RCA) line level switch boxes. And Manley is making the Skipjack line level switch again, as I recall.
Look, no one would put a switch box in the signal path for no reason. But if you need to compare gear it's a much better way than powering up and down and swapping cables.