"I guess I was under the impression that the drivers were the most costly and contributes mostly to how speakers sound. Given that revelation couldn't you use cheaper drivers with a great cabinet to get close to the same sound as better drivers?"
Maybe. In practice, most designs have to compromise somewhere in order to provide an affordable yet quality product.
But in the end its how all the component parts of a "speaker system" contribute together to deliver the sound people want that matters.
Lets not forget that they are "speaker systems" after all, comprised of various parts, often but not always including cabinet. Some may have no cabinet at all as pointed out.
Its like making soup. There is margin for error (nothing is perfect) but get any one ingredient in the soup wrong by enough and the taste will suffer.
But also remember the speaker system alone is still not enough to guarantee good sound. There's how well the speaker works located in your listening venue/room and how well the amp is able to drive the speakers. Only once all that is set up well are you in a good position to judge how good your sources (CD, phono, etc.) really are. After that, your entire system is tuned in which is what it takes t get the best sound out of what you can afford.