noromance has it more or less. Bypassing caps has it's basis in good EE design, especially older but also best bang for the buck designs.
Think of actual caps as a series of parts:
--> Capacitor --> Inductor --> Resistor --->
As the frequency goes up the cap no longer acts like a perfect cap and the inductance becomes a problem. By bypassing this with the appropriate values of smaller, higher quality caps you can get maximum value for your money. Imagine a situation when you need 100uF. The cheapest option may be an electrolytic but may become inductive too soon. So you buy a 100uF and bypass it with something pricier (per uF) but smaller, like a 2uF film or tantalum cap to overcome the internal inductance.
The reality of audio gear however is complicated. Sometimes a good bypass cap can help, sometimes it does nothing. I like Clarity CMR caps a great deal, but at values larger than 4uF or so they do audibly benefit from a small bypass cap. Below this value I've not found a bypass cap to help at all.
I suggest you get a pair of Audyn TruCopper 0.1 uF film caps and experiment for yourself. The argument of the value of bypassing or not is not ever going to be solved in an online forum. :)