Yes you do need a good quiet power (toriodal) supply that keeps noise away from the circuitry - and big capicitance like on a big power amp is one way to do that. Of enormous criticality is how power supply is kept away from ground and how balanced the signal lines are designed with respect to ground.However, it can't be simplified to just that.
It is just as important to have features other than just a good power supply - balanced, high output levels with great channel separation and low output impedance and high input imepdance and of course low S/N. High quality tone controls with a "defeat" option for refernce. Volume control needs to be well implemented like with a discrete resistor network steps (like a crystal CS3310 or Burr Brown PGA2310) rather than just a cheap pot and wiper with continuous volume control, IMHO.