I agree with ghdprenice , it's all relative to your system and future goals. That holds true no matter the component. I am a regular guy, there is no way I could assemble a great system overnight . That said every upgrade should be more than subtle , or maybe that's not the component that should be replaced.
Best example for me is my Zesto preamp. That WAS the right component to buy at the time because the difference was so pronounced and appreciable by anyone who heard it . Could I afford it new? No. But I got a great deal on a demo so it was worth every penny. It will be my last pre for a while because in my system my preamp was the bottleneck. Now any upgrades moving forward are fully realized.
Today chipsets are very , very good. It's the way it's implemented and really at the end of the day how it sounds in your system. A good dealer is invaluable and will let you take a demo unit home to evaluate. That is my preferred way of upgrading. A lot less trial and error.