Should I get a new cart, TT or phono stage?

At some point i want to replace all three but I'm wondering what will make the biggest improvement first.

I have an old Denon DP-45F with a recently purchased Ortofon 2M Red going into a Pro-Ject Phono Box II which then goes into a Simaudio i1 integrated.

What would you upgrade first. Some things i was thinking of were a Rega P3, Clearaudio Emotion, Simaudio LP3 or used LP 5.3, or a ortofon 2M Black.

If I made any upgrade i would likely be stuck with the remaining components for some time before I made the next upgrade.
Keep the 45F. The difference between a 45F and a Rega is not worth the loss of all the automatic features of the 45F, assuming your set up of the 45F is optimal. I would change the phono preamp to get a better match with your cartridge. I have a 47F running to a Marantz SC9 and also a VPI Aries. Yes, the Aries is better since it has a $2K cartridge and SDS box, but the 47F is also great and it has all the auto features.
Worth taking a look at the entry-level SOTA tables while you're at it. I write from experience with the Rega, SOTAs, Music Halls, Micro Seiki, and Pioneer but not the Denon tables.

That's the tough part for me I don't know what I'm not hearing. I'm just looking to make improvements.

The turntable sits on vibracones and it has built in interconnects. The cart was installed by a dealer and I assumed it was done properly. The turntable is level and I don't think I'm getting any feedback.

Overall I pretty amazed at how good the denon/ortofon sounds, but I'm wondering if an upgrade of either will make a noticeable difference.
You didn't specify the amount of money involved but IMO it is the phono preamp by default.

First, the turntables you mentioned are no better than the Denon. At best you're going sideways there.

Second, the Black is a bit better than the Red but in the Denon's arm the difference(s) will be marginal at best.

Go for the phono stage. A pre-owned Graham Slee Era Gold or comparable product would be a nice upgrade for a minimal investment.

You might also try some isolation devices on the Denon.

Good luck.
what you must consider is how far you will up grade? if you are going to move only a notch or 2, there will probably be very little imporvement. if you are going to move up 5 or 6 notches(out of 10) then the difference will be much more apparent, although, it will be dependant on your weekest link. if you buy a top of the line clearaudio tt but keep the same cart an/or phono stage the improvement will be limitted to to their ability and so forth up the chain. my advise would be if you are going to do it make sure you are committed to seeing it through, and move up the latter the furthest you feel you can comfortably afford. in the end, which probably there never will be, you will be glad you did. Vinyl, not just music, but a way of life.