How cheap can you go and still be happy with the sound?

Knowing what I know now, I could probably live sub $1k.

I recently subbed out my Modwright PH9.0 with a vintage Moth Audio phono stage ($200) and it sounds pretty good!

I also picked up a Dual 1218 for $60. I put a V15 typeII on it.

Add a decent receiver and I could be happy.

I think though, you have to experience really good first, so at least you have a baseline; you have to know what minimum makes you happy.

I'm thrilled with my PSB Image 2B front and Image 2C center channel speakers, REL subwoofer, and Radio Shack Optimus LX5 rear channel speakers powered by my Yamaha RX-V571 receiver(soon to be replaced with an Aventage RXA-4A receiver)! Even if I had a massive reserve of disposable income, spending hundreds of thousands of dollars on HiFi/home theater components is unnecessary and ain't happening!!! But, I'm not hating on those who have the resources and choose to do so, 

DIY speakers.  Lots of really excellent designs.

Onkyo C-7030 CD player

Good used DAC (optional)

Monoprice Stereo Hybrid Tube Amplifier 50watt  (integrated)

Smithwire 12ga stranded wire for speaker cable (Home Depot, Lowe's, etc.)

Find interconnects to your liking.  

Some klipsch rb62 II and an alan eaton amp. Setup under 2500

I've owned both ends of a HiFi budget, Nothing beyond Pass, Esoteric, B&W, PS Audio... And I can assure you, hi quality HiFi can be had for a bargain if you do your homework. The 'cheapest' isn't necessarily a bargain in the long run.

Here is an example. The Roon Nucleus, a very popular server, runs around $1400  USD. I purchased a preowned i7 NUC, 3.4 Ghz quad processor... Blah, blah for $120 on eBay. You can do the same for DAC's, Amp's, Transports... 

The flip side is pride of ownership. I remember a fellow right here many years ago was school teacher (low pay scale). He had cobbled together his system over three decades and always wanted an ARC preamp. He finally purchased one. He said thats the reason I drive a 20 year old Lexus lol...