No should be fine, as it will be a nice low low output impedance because of a headphone driver opamp, but a normal output with VC would probably sound better because the head phone opamp won't be in the signal path.
Just turn the VC down before you try.
Cheers George