Great integrated amps are available on the used market for under $2,000. Various models from Musical Fidelity, Plinius, YBA, Classe, Conrad Johnson and Manley are all readily available. If you stretch your budget to $2,500 you might be able to find an original model Rowland Concentra.
The choice for loudspeakers really depends on your listening preferences and your room. If you have the space and aren't into "power" music, than the Quad 63s are outstanding. They will also mate very well with any of the above integrated amps. They're readily available for under $2,000. The Quads will provide state of the art midrange and imaging with excellent tonality. If space is an issue then go with the Proac Response 2s. If you taste tend more towards loud music with lots of bass, then the NHT 3.3s are a good choice. They can be had for $1,500. The NHTs have terrific bass, excellent imaging and strong dynamics.
An Esoteric (Teac) DV50 universal player can be had used for $4,000. Alternatively, you could go the hard disk playback route. An Apple G4 laptop ($1,100), an Apogee Mini-DAC w/ USB and an external firewire 180 Gb hard drive ($300) would make an excellent sounding digital playback system when using Apple's iTunes software. Add and iPod ($300) and you'll have portability. The Esoteric would provide the better sound quality, but the Apple hard disk system, while still excellent sounding, provides far greater listening flexibility.
Used cables are an excellent bargain. I prefer neutral type (Cardas, Discovery, etc.). At this system price level I wouldn't recommend spending more than $500 on all cables.
Good luck!