I adjust arm height (and thus VTA/SRA) for every LP. This is because I hear the differences and appreciate optimal sonics from each record, not because I love tweaking.
Not everyone is on the lunatic fringe. Depending on your tonearm, your system and your sensitivity, you may want to do that much... or much less.
I'm happy to explain my particular methods, but if you're more inclined to set and forget it wouldn't be helpful. Can you clarify your equipment (e.g., list your System), musical preferences and sonic goals? That would help others respond in ways appropriate to your particular needs.