What is the most difficult component to upgrade

I tend to agree with you. I had the same problem with my pre-amp. It's odd, what should a pre-amp do: select sources, control the volume (pretty simple). What should it not do: compress, roll-off, degrade, or otherwise color the sound. With such simple goals (the best pre-amp is no pre-amp kind of philosophy) you would think selecting a pre-amp would be easy. But like you I found one to have special qualities in one area and lacking in another. I'm quite happy with my current pre-amp, but I'm also very certain it is not "the one".

I find Speakers are ultimately the hardest component to upgrade.

Why is this the case?
Because speakers vary more in presentation than perhaps any other component. They are the only piece that converts energy into a different form in a pure digital setup (a cartridge does do it as well in an analogue setup).

I try to keep speakers as long as possible, and build a system around them. Some speakers are easier to get to sound good than others. Some are hard loads to drive, others are easy loads. Some need power becasue they are not sensitive, some are way sensitive and need less than 1 watt of power for good volume output.

In any case, the greatest system with one set of speakers may not be near the greatest system with another.

Peace.

KF

For me, it is the speakers. I've tried twice and kept my old ones that cost about 1/2 to 1/3 of the price of the ones I tried to demo at home as an upgrade.
I think it is a case of, I've done such a good job integrating and tuning/tweeking my system, that a speaker change throws it all out of wack.
Based solely on price point my speakers should be the weak link, but they have no weaknesses that could have been revealed by the better electronics.

Strictly speaking I think the room is the most difficult. The present state of affairs makes finding a digital source to be most challanging.