What is downstream (speakers, amps) must be transparent enough to let what's upstream (source) through, else you'll never be able to hear a valid difference.
So, work downstream first - you'll get the biggest bang for the buck.
Later, higher-end source components, will give you increased and more accurate detail while decreasing the harshness associated with digital -- closer and closer to "natural" uncolored sound as you and your peers "hear" it.
