ITball,
You are right on for the most part. Speakers, and their interaction with the acoustic environment (room) accounts for 95% of the sound you hear, good or bad, assuming that the rest of the components are at least mid-level quality.
Tubes vs. Tranistors is another thing that you will hear, though this might require a bit of practice and a revealing system. A 40W push-pull EL34 amp like a Jolida will sound pretty different than a Class A Krell or something. With a well setup system, this is something that everyone should be able to hear. Not to say that all tube amps/transistor amps sound different, but some topologies (like the above) will.
Tied with that, changes in source are pretty audible, especially with analog. Listen to a Rega table with a cheap cartridge, and then a big VPI table with an expensive cartridge. You will hear some major differences here.
Digital sources are mixed bag, and changes are often quite a bit more subtle. Like one of the above posters said, a higher-end digital source can be the difference between being able to easily follow a bassline, or having it kind of getting lost in the mix. It won't be a major "this whole stereo sounds like crap" thing with a cheaper digital player, unless the CDP is really bad. (1989 Circuit City Sony or something)
Like you, I don't hear notable differences between cables, tubes, most tweaks, and SS amps.