Lewm, to be technically correct:
In a single-ended setup, in which (in each channel) one of the pins of the cartridge is connected to ground, the phono stage amplifies the signal present at the other pin of the cartridge with reference to ground. In a balanced setup, in which none of the pins of the cartridge is connected to ground, the phono stage amplifies the difference signal between the two pins of the cartridge; no reference to ground is present. That's the advantage of going balanced: No reference to ground with all the noise, grunge... present there.