In doing that you are of course connecting the two outputs to each other, as well as to the desired input. And if only one of the two source components is turned on at any given time you are loading the output that is being used with whatever output impedance the other component presents when it is turned off. Depending on the specific design that could be just about anything.
In some cases the unpowered component may have a muting relay at its output, which would put a direct short on the signal being provided by the other component, resulting in no sound. In other cases the output impedance of the unpowered component may vary with the level of the signal being provided by the component that is being used, resulting in poor sonics at best. And I wouldn't rule out the possibility that there may be a few cases in which damage could occur, eventually if not sooner, as a result of a short or near-short being applied to a powered up output.
The suggestion by Donvito is a good one. Another possibility would be a line-level switchbox. I have used the DB Systems DBP-2JAU/5 ($110) with good results. See the "photo" and "review" links for the DBP-2J/5 near the top of the following page (that model is the same as the DBP-2JAU/5 except its connectors are not gold plated). I've also provided a link to their home page, at which you can find contact information:
https://www.dbsystemsaudio.com/access.html
https://www.dbsystemsaudio.com/
Also, at a considerably higher price point Decware makes a nice switchbox:
http://www.decware.com/newsite/ZSB.html
Regards,
-- Al