The jack ain't the issue; it's what's driving the jack. You preamp has a relatively low output impedance and does a fair job of driving the phones. However, the fact that you have to set the VC at 12 o'clock suggests that there's not much power output available.
A dedicated phone stage, regardless of the output jack, will have more power and a lower impedance. The result will probably be lower distortion.
A dedicated phone stage, regardless of the output jack, will have more power and a lower impedance. The result will probably be lower distortion.