Help Me Build a Vinyl System

The man wants to try vinyl and has some cash to do it right. Good idea. There are millions of records around, that's not a problem. Cleaning is not a problem either with a $600 machine. There are no big problems with vinyl, just some playing with the set-up. The difference with well recorded classical and jazz compared to digital should be huge.
For $5000 used there are so many options. What kind of sound are you looking for?

Go for it. Get a VPI.

Here's my recent experience - I started from relatively zero and now have a fantastic vinyl playback system and decent record collection for under $5k (used VPI Scoutmaster, Soundsmith Zephyr, Sutherland Ph3D.)

Vinyl's gotten me both amazing sound quality and tons of fun.

You can get mixed lots of 100 classical LPs here on Audiogon for about $2/record, $5 - 10 for fancy labels. Mixed bag, but part of the fun is exploring what's there. I'd never pay $20 each for 4 different recordings on CD of the same piece. But, its easy and great on vinyl. On the other hand, the high quality reissues - Music Matters Jazz, Acoustic Sounds, Mosaic, etc - are simply amazing quality, though expensive ($50 each)

Let's forget your statement "...to start with"
You gonna get it right once and for ever!
1) : Lenco L75 (less than $200 ebay) + (about $350 ebay) plinth with removable armboard (jec965)
2) : Zeta tonearm ($900 Audiogon)
3) : Denon DL-103R Panzerholz uwe body soundsmith level 1 ($375 Audiogon)
4) : Einstein TT CHOISE phonostage ($2650 Audiogon)

Ps : You must be very quick to grab 2) & 3) ASAP !

Lots of good sound advice.. Rauliruegas spelled it out perfectly...I started in the same position as you are now.
I have spent approx 15K on my vinyl system and lps in just over a year. My budget was 5K...good luck!
I hate cleaning records too!

Correction: Jazdoc had it spot on with help from Raul.