It always surprises me that most people seem to agree with the notion that speakers make the biggest difference in the sound that is heard, yet at the same time many of those same people consider various upstream components to be more "important."
As to the original question, while good points are made in several of the preceding answers, I particularly agree with the response by RLWainwright.
Regards,
-- Al