The main problem might be a possible overload of the phono section or the preamp input section. You would hear the distortion if overload was taking place. Especially on high peak passages.
If you are hearing no audible distortion on peaks, then there is likely no overload(or very little) taking place. If you hear it, then there is.
Increased distortion can possibly cause damage when amplified at higher levels into the speakers.