First, if your LP's and stylus are not perfectly clean, inner groove distortion (IGD) will drastically increase. As you're still relatively inexperienced with LP playback at this level I'm fairly confident that your LP's and stylus are not perfectly clean. My Audiogon buddies with many more years of experience have seen the extent I go to in this regard and they've heard the difference it makes. I've re-cleaned records of theirs and given them sonic revelations. This is job 1, the best system and setup in the world will sound like crap if crap is what it's playing. ;-)
***
As to technicalities, assuming perfect LP and stylus hygiene...
Your description does sound like IGD... tormented vocals and acoustic instruments, particularly in the higher registers. IGD can make massed violins or a soprano sound like 10,000 fingernails on a chalkboard. Some LP's are more prone than others, depending on the material that's cut on the inner grooves.
The root cause of IGD is the unavoidable fact that inner grooves are shorter than outer ones. Each LP groove makes one complete revolution in the same length of time (1/333... of a minute) but since the circumference of an inner groove is shorter, the modulations that make up any given frequency are compressed more tightly than they would be on an outer groove.
It's harder for a stylus to trace these smaller, more closely spaced modulations. This is why alignment is critical. The key parameter is zenith angle (what we adjust with a protractor). For this reason, I second Dconsmack's excellent post.
Zenith angle is critical for reducing IGD, other parameters are less so. VTF, VTA, azimuth and antiskating have comparatively little impact on IGD unless badly mis-adjusted.
***
The capabilities of a phono stage also have a major impact. Poorly modulated phono stages go into peak overload and/or exhibit smearing distortions that can greatly exaggerate the audibility of IGD. A better phono stage can make a huge improvement in this area.
***
BTW, after a certain point IGD is unavoidable. Consider: at any given point on a groove's inward spiral there will be *some* frequency whose modulation has the same radius as the contact surface of the stylus. Adjust as we may, our stylus cannot physically trace a modulation smaller than itself, which means it cannot accurately reproduce any higher frequency than that. Instead, it will "smear" over the tops of the waveforms and produce a distorted version of the actual sound.
As the groove spirals inward and its modulations get smaller, the highest frequency any given stylus can reproduce drops lower and lower. On widely spaced outer grooves even the coarsest stylus can reproduce very high frequencies. On tight inner grooves even the finest playback stylus cannot match the smallest modulations cut by the (even finer) cutting stylus.
Nevertheless, perfect zenith alignment at the most challenging part of the record helps because it places the L and R contact patches of the stylus squarely across the groove. This keeps the difficult-to-trace higher frequencies precisely in synch across the L and R channels, reducing phase errors between them. This helps reduce the "nails-on-chalkboard" effect.
***
From the above it should be obvious that the smaller the contact radius of the stylus, the less prone it will be to IGD. Line contact and micro-ridge styli are the best, conical/spherical styli the worst. Ellipticals fall in between.
***
The manner in which a particular cartridge/tonearm/table combination handle stray energies leaking from the cartridge also has a major impact on how audible IGD will be. Some combos are very good together. Others are terrible. There is no set of specifications that will predict this.
