You are right on, @karl_desch - tiny changes to azimuth are easily measurable and audible. Of SRA, VTA (different things and important for different reasons, actually!), zenith correction and azimuth, the azimuth angle is most sensitive to angular changes. This is not only borne out subjectively, but there is a mechanical reason to it as well.
If more people understood the purpose of the stylus’ major radius, perhaps they’d be less intent on visually aligning the azimuth angle and pay attention to what really matters - an orthogonal relationship between the 45 degree groove wall and the coils (discounting any left/right variation in the magnetic flux field, of course).
Below is a link to an image of a PROPERLY aligned cartridge on the azimuth axis. 2.6 degree tilt! While this is the product of less than attentive assembly, it is certainly where this cartridge performs its best. Its ~50um major radius allows for this kind of angle without causing one contact edge of the stylus to meet the record land/groove junction. Both contact edges remain properly seated in the groove.