Adjusting azimuth affects crosstalk and relative phase of the two channels.
People can accept some level of crosstalk, particularly from speakers since you're hearing each speaker with both ears, but many cartridges (particularly less expensive ones) may have crosstalk in the -20dB range which isn't great. Getting crosstalk to at least -30dB is better. A digital system will usually have crosstalk better than -80dB across the entire frequency response. And I've read that -40dB is about the threshold at which people don't really care much or notice anymore. (Not sure how correct that number is.)
But more egregious phase differences can really mess up the sound. If you imagine a singer but their voice is coming out of your left and right speakers are slightly different times, it will sound really off and bad. And that is what will happen with all of the instruments.
That being said, you need to correct azimuth and phase electrically or using software. The actual physical location that needs to be calibrated is inside the cartridge, where the magnet and coils are, and their alignment is not perfect with the cantilever. Small 0.25° of change in the azimuth make a difference in the measured crosstalk and phase. But that's a very difficult angle of change to try and measure by sight. And every cartridge's internal alignment is slightly different, even of the same make and model.
This is why paying for a professional calibration can be worth it. You can of course learn how to do it yourself and purchase the necessary tools and/of software. Alternatively, don't worry about it too much and just enjoy your music if it already sounds good to you.