Maybe I missed it but what power amplifier are you using? Aside from the general tube "warmth", what you might be experiencing is an impedance mismatch between the preamp output and the amplifier input and engaging the tube buffer could possibly correct this potential mismatch. Do you know what is the preamp output impedance with and without the tube buffer and the amplifier input impedance?
Just a thought...
Just a thought...