I don’t think you need to spend $6k to get to a much better place. It’s possible you can spend double that and end up with terribly mismatched components. I think you can do it for less than $4k.
But, yes, spending $6k+ can make it less challenging for sure.
So if you do decide to pause to save some money, I would at least, for the time being, get a better MM cart. Your Elyse 2 is an elliptical. It’s a fine cart but you can do better. Try a good micro line or shibata stylus. I can recommend the AT VM740ML or 750SH. If your phono stage can do 100pF capacitance, you’ll really enjoy the added detail and dynamics these stylus profiles can provide.
These carts are the current successors to the famed AT150MLX. What’s great is that you can start with the 740ML (micro line) and replace the stylus with a 750SH (Shibata) when time comes. You can even upgrade to the 760SLC (Super Line Contact) as they all use the same engine/body.