Grbluen2,
There is no procedure for finding the 'magic setting', as every motor is different. For example, I have three Hurst 300 RPM motors, but really only one is worth a damn - the other two can never be quieted down to a suitable level, and never quite sound right. I set a motor so as to null vibration as much as possible - you can either hold the motor in your hand, use a mechanics stethoscope, or listen to the table rotating in a lead-out or locked groove. From there, I'll look at the waveform on an oscilloscope to ensure the sine waves look nice; I generally find that a slightly distorted waveform actually yields a better result than a pure sine wave (I don't know why). From there on, it's all in the listening, trying to balance torque requirements with vibration considerations. I'll vary the voltage of each phase slightly, and make assessments from there. For example, a motor with too much voltage will always place too much emphasis on the attack of notes, while too little will yield a bloated bass and a really laid back presentation. The more you play around, the easier it becomes to hear where you're at and establish a baseline sonic signature - speed never varies given the nature of the motor, but the way the motor deals with heavy modulation, stylus drag and the like can all be readily discerned. Unfortunately, there is no purely objective way of measuring any of this, like so much in this odd little hobby. Trust your ears, and beyond all else, have some fun.