My 2-cents worth. My Rega RP3 -- bought about 12 years ago with a Rega Elys, upgraded 7 years ago to a Dynavector 10x5 -- gives me better sound than my CD Player (NAD C516BEE). I upgraded 2 years ago to a better phone stage (Pro-Ject Tube Box DS), and got significant improvement.
It's a mood thing. When I am working or just need background music, I play CDs. When I'm really listening, I play LPs.
And no, my analog rig was well within the $1K when I started. I added a few bucks every few years. So NO -- you don'r need big bucks to enjoy vinyl!!
G