If you are experiencing lateral drift when you drop the needle on the record, either to the left or right, check your antiskating. Ensure that you have mounted the LPS correctly. Though not a Phantom, I have mounted several LP's on SME V arms with no issues. The Phantom has more user adjustability, therfore, you may have to pay very close attention to each and every setting on the Phantom arm.
Good luck.
Audioquest4life
Good luck.
Audioquest4life