I couldn't hang with a headphone only system. Even in urban areas there are the suburbs where you can listen to music in the privacy of your own home.
That said, I'd propose a compromise: down size to a 2.0 or 2.1 system with a quality integrated. There are many, but Musical Fidelity makes quality integrateds at good values.
So I'd go with a thinning out verses throwing out. Less to move, easier to setup, but you can still enjoy the sound of music in your house/apt. And you may even end up cash positive :)