No traditional pot meter is particularly accurate at low levels. Does it happen with all sources? I ask because there may be a mismatch between the output sensitivity of the source and the input sensitivity of the amplifier. Many modern sources like disc players or DACs have the nominal 2.0V output of the CD Red Book standard, but some have even more (to impress in the demo room - louder sounds better). At the same time traditional rca inputs are far more sensitive. This creates two problems. The first is that, depending on the amplifier topology (where is the volume control, before or after the input stage?) the input stage may be clipping from the high signal (with all the sonic issues that come with that). The second problem is that if you want to play at a comfortably low level, you have to turn the volume control down into the range where the pot meter is not very accurate, and you get channel imbalance. Do you get a pretty loud signal already very quickly, i.e. when you turn up the volume control perhaps only a quarter? If that is the case, you need to reduce the input signal. Inline attenuators are the simple solution.
This is not to say that the pot meter may not be failing as well. But even if it is no longer at its best, inline attenautors (or an attenuating cable) may still be a cheap fix (Partsexpress has attenuators from Harrison Labs).
This is not to say that the pot meter may not be failing as well. But even if it is no longer at its best, inline attenautors (or an attenuating cable) may still be a cheap fix (Partsexpress has attenuators from Harrison Labs).