This might be a stretch as you may have already tried this, but it's worth a shot.
The manual for this unit mentions it has "variable analog output". Since the attenuation is probably done in the digital domain I'd run it at full "volume" into the preamp so as to not drop any bits. Check that this setting is not the cause by hooking it up to the preamp, run the "variable analog output" up to full and use the preamp's line stage to control the volume.
FWIW, running it straight into an amp would not test be an adequate test unless it was done at full output and then what could one tell anyway?
Got my fingers crossed for you...