Capacitors have their own technology, dynamics and very often audible characyeristics. Anybody who disagree most likely never replaced one in the audio path of their high-quality equipment piece.
As for capacitor used to shunt DC power supply, they are necessary to eliminate high-frequency noise present there, but never affect the sound quality as much as a different coupling (serial) capacitor used in the audio path.