IME, the most satisfying low-level listening is headphone listening. It’s not even close and that would be at a very modest level of equipment. YMMV.
With speakers, you may want to look at ways to create your own loudness control. If you are doing computer audio, this might be software-based equalization. For example, I have had some success experimenting with Roon’s Parametric EQ. Or you could get some kind of physical equalizer box. In my very limited awareness of those options, Decware’s ZROCK would be at the top of my shortlist.
FWIW, when I have had the opportunity to have some casual q+a with a couple of different speaker builders/designers on this topic, efficiency did seem to be a factor in their recommendations.
Good luck!