Unclejeff is incorrect here. Usually in a balanced amplifier the RCA corresponds to either pin one or pin three of the XLR.
The real issue is that the loading from the two sources will interfere with each other and the sound will be less than optimal (rolled off, distorted or both), although there is no risk of damage. So you will have to continue switching cables unless you can convince someone to install a switch for you.
The real issue is that the loading from the two sources will interfere with each other and the sound will be less than optimal (rolled off, distorted or both), although there is no risk of damage. So you will have to continue switching cables unless you can convince someone to install a switch for you.