The problem is not your amplifier. I had the same experience with Totem Model Ones. They needed a good amount of volume from my Krell KSA 80B for the sound to become balanced.
When Robert Reina reviewed the Cayin A-50T he reviewed it with Monitor Audio Silver RS6 speakers. He said "the Cayin's midrange reproduction was that of an amplifier able to resolve an extraordinary amount of inner detail". "With every recording I played, the highs were pure, extended, airy and delicate, with no trace of coloration."
This detail is what you are missing with the Totem speakers at low volume.
Maybe you should try the Monitor Audio Silver RS6 speakers. They seem to be a good match with your amp.
When Robert Reina reviewed the Cayin A-50T he reviewed it with Monitor Audio Silver RS6 speakers. He said "the Cayin's midrange reproduction was that of an amplifier able to resolve an extraordinary amount of inner detail". "With every recording I played, the highs were pure, extended, airy and delicate, with no trace of coloration."
This detail is what you are missing with the Totem speakers at low volume.
Maybe you should try the Monitor Audio Silver RS6 speakers. They seem to be a good match with your amp.