Fish, until you make a change to try a more effective pairing of speaker and amp, not sure there is much else to say.
Once you have that in place, then additional tweaking may or may not be needed as well from there. IT all depends on how "finicky" you are able getting a particular sound that you seek. Its the typical scenario for any "audiophile" . It may take some time and money but if enjoying your music to the max is your goal, it should be worth it in the end.