No.
Every contact, additional component and wire/trace affects the
sound. Anything below -20dB is pointless with properly configured gain
structure.
Is it HiFi or sonic wallpaper?
If it is contact less already on the signal path then you can have as many inputs as required, and zero volume too.
To do that arrange LDR anodes of each series pair to be switched on in turn - according with your stereo inputs. To zero volume needs your design to power shunt anode separate to series anode, it needs also NOT to ground the LDR cathodes, rather arrange a regulated jfet current sink.
Your volume control is arranged between shunt cathode and series cathode. You then arrange via LM317, with a higher potential on its adjustment terminal, place its Vin to the series cathodes and adjust for silence with resistance from the higher potential to Adj. As lower volume positions are reached, the start circuit thus named influences the series resistance readings, and causes extreme resistance on series pair at lowest volume, creating ability for silence.
The start circuit is accompanied with another circuit called the continue together they interact together to improve relationship of the entire circuit for excellent current delivery and resulting audio performance.