Just my 2 cents, I would focus on the source first before anything else. Of course if the amp is weak than that should be taken care of, other than that, one will never know what the system is capable of until the source is optimized.
From my experience, I didn't know what my speakers were capable of until I got my vinyl rig greatly improved!