Is that the same NC500 you bypassed the buffer clearly violating the recommendations of the app note that the amplifier stage be driven by a low impedance source to maintain performance?
There is also capacitance in that input which means that a high impedance input will cause phase shift and a non flat frequency response.
2.5.1 Audio input
The INH/INC inputs form a differential pair. Note that the input impedance is fairly low meaning that
minimalist discrete circuits or valve input stages won’t work. All op amps commonly used in audio can handle
them though. See section 9 for suggested circuits and connections.
Do not drive the input with fully floating sources, be it electrically floating ones like line driver IC’s intended for driving XLR outputs or transformers. Using a floating source will always result in a common mode
component that will exceed the common mode input range and will manifest itself as gross distortion.
Make sure to set the outputs of your distortion analyser to grounded, not floating.
There is also capacitance in that input which means that a high impedance input will cause phase shift and a non flat frequency response.