It’s caused by suboptimal gain structure. You have way too much active gain after the volume control (this means your combined preamp and amplifier circuitry - which generates higher noise floor per extra dB of gain even if signal-to-noise remains the same), relative to your needs determined by listening level and speaker sensitivity.
I have this situation myself, with very sensitive 96dB / 2.83V Tannoys. I easily notice the signal-to-noise ratio differences between preamps. I need to keep the preamp gain below 14dB, and even at that level the preamp’s signal-to-noise ratio has to be VERY high to avoid an audible "hisssssss" at volume = 0. The ARC Reference 6 has a stellar signal-to-noise. VAC Master also excellent. Some other preamps...are not so great.
Often inserting signal attenuators (e.g. Harrison Labs) to the tube of ~ 10dB is recommended to push the noise floor down while utilizing more gain from your circuitry. However, if you have the Parasound amps, as @russ69 mentioned - turning down the gain pots should be the best solution here.