In my opinion, it’s about matching speaker sensitivity to the amp. That said, I have found that “higher power” amps, especially solid state, you have to search for the sweet spot on the volume knob. That point where every instrument is playing at its best. I currently drive high sensitivity speakers with 6 watts of set power and It is perfection, at least to my ears. Not just at the sweet spot but exactly what you are experiencing, at lower volume levels. If you are thinking about a less efficient speaker, something like a maggie you will need lots more current and watts. Had some 3.5r’s and found them as sensitive as a rock, they needed lots of juice. Sounded great but needed the proper amp. If you are going to stay with what you have, for a while, don’t rock the boat. That said if your next step is falling for a different speaker, no matter the efficiency, cross that bridge when you get to it.