How cheap can you go and still be happy with the sound?

DIY speakers.  Lots of really excellent designs.

Onkyo C-7030 CD player

Good used DAC (optional)

Monoprice Stereo Hybrid Tube Amplifier 50watt  (integrated)

Smithwire 12ga stranded wire for speaker cable (Home Depot, Lowe's, etc.)

Find interconnects to your liking.  

Some klipsch rb62 II and an alan eaton amp. Setup under 2500

I've owned both ends of a HiFi budget, Nothing beyond Pass, Esoteric, B&W, PS Audio... And I can assure you, hi quality HiFi can be had for a bargain if you do your homework. The 'cheapest' isn't necessarily a bargain in the long run.

Here is an example. The Roon Nucleus, a very popular server, runs around $1400  USD. I purchased a preowned i7 NUC, 3.4 Ghz quad processor... Blah, blah for $120 on eBay. You can do the same for DAC's, Amp's, Transports... 

The flip side is pride of ownership. I remember a fellow right here many years ago was school teacher (low pay scale). He had cobbled together his system over three decades and always wanted an ARC preamp. He finally purchased one. He said thats the reason I drive a 20 year old Lexus lol...

I think anything can move me .

Perhaps another sample from the used purchases category that I regard as great value some are not low budget…

Audio Research D240 MKII - original cost USD$900

Audio Research LS10 - original cost $1100

Audio Research Ref CD9- original cost $6000

This is a complete set up. Just add speakers.

The CD9 doubles as a DAC for any streamer source (I use the Cambridge MXN10). It’s the best DAC I’ve ever heard.

I know this can’t possibly fit this brief. 
Today I’ve lost the LS10, it blew its power module. I replaced with an Marantz AV8805 that I have on hand. It sounds incredible playing through the D240 MKII! It’s such a music amplifier.

Sorry for this high quality interlude. It’s an avenue.