My suggestion would be to look at a used VPI Classic 1 with a JMW arm, which tend to hit the market used for around $2K or less. It is truly a high-performer with a 20lb aluminum platter and very heavy plinth. The tonearm tracks very well, and the whole turntable can be upgraded - tonearms, footers, clamps, etc.
For cartridges, I would look for an Audio Technica ART9 xi. Best cartridge around $1K hands down, the only cartridges I liked after owning the ART9 cost $5K+. This though would require you to have a moving coil phono stage.
For phono and amp, if you do not look into a speaker amplifier, then look into the Pro-Ject Tube Box DS2 and Woo Audio WA7 fireflies which is an incredible tube headphone amp with a great built in DAC.
If you play your cards right you should be able to get all this for less than $5K, but you may need to leave a little extra for quality cables as these components will absolutely require them for optimal performance.