Your headphones' impedance could be too low even all by itself. Many headphones have impedances like speakers', in the 4 to 16 ohm range. (Putting a high impedance amp in parallel to a pair of headphones would not make a lot of difference.)
Presumably, you would unplug the headphones while listening through your speakers, so the preamp should be OK then. But when listening through the headphones, the low impedance of the headphones could result in damanding more current from your preamp than it can supply without distortion, as others have already aluded.
So, why don't you plug your headphones into the output of your amp instead?