I'm in a similar position. Currently have Wharfdale Jade 3 bookshelf speakers and have been considering Harbeth P3's. The Jade 3's are really nice in my small room but the P3's shine in the mid-range female vocals that I prefer. Is it worth switching? Don't know. I may have to buy some P3's and live with them for a few months.
Driving the Jade 3's with a Cambridge CXA-81. Better to keep the Jade's and move to a better amp or DAC? Just don't know. PM me if you decide to keep the Totems.