The amplifier is the most important and by a wide margin. Why? Based on my experience, most amplifiers do much more than just amplify the signal. And that's almost never a good thing. In addition, it appears that most amplifiers will only excel in one or two parts (if at all) of the frequency spectrum (bass, mid, or treble) and not do so well in the other part(s).
In addition, I'm a firm believer that a great amp can made bad speakers sound very good. But never will a great pair of speakers sound good when paired with a bad amp.
I believe that there are more character differences and greater deficiencies amongst amplifiers than any other component.
-IMO