Preamp Gain issues with high efficiency speakers

Hey all,

Been having a hard time with my system lately.

I recently got a pair of Zu Soul 6 that are 100db efficient. My power amp is a Conrad Johnson Premier 11 (70wpc) and my Preamp is a Conrad Johnson ET3-SE that has 25db of gain. Source is a streamer (PI2AES) with Denafrips Pontus II DAC, but it is the same regardless of source.

My issue is that even at 1 out of 100 on my preamp, the volume is still quite loud. If I were to turn it up to 30, I would be well north of 95db.

Due to a new baby, most of my listening is at low volumes.

I have been using in line attenuators but I feel like it adds a slight veil to the sound of the music.

I swapped out my Preamp for a Schit Sys passive pre and I could only turn the knob to maybe 7 or 8 pm before it was LOUD. I figured with a passive no gain pre I would be able to have more control over the volume, but this was not the case.

Is this Amp just a bad match for my speakers? I have read that it is better to have your pre running closer to 40-70% so that you are not attenuating the signal too much but I can barely get it to 5-10% before its quite loud.

Any insight, recommendations would be appreciated!