Actually, I believe we do agree here. There are several rather precise measurements and settings that must first be made visually, which if done correctly should set the analog deck up correctly - which is to say, it complies with the mfgs design and intent for the arm, table, cartridge. Fine tuning to compensate for less than accurately made settings, incorrect parameters provided by the mfg, wear and tear on the TT, etc., and to satisfy your own desires for sound should come after that.
I fail to understand how one could possibly set up the TT and cartridge using ears alone - there are simply too many variables. Seems to me that the vta, vtf, overhang, mounting distance, etc. are all set up using your eyes (at least initially). Which brings me back to the start of the thread - how to visually set the antiskate close, if not spot on. After these adjustments are set close enough, then fine tuning by ear could begin.