Lighter materials are inherently faster and more re-active
Inuitively it would seem that way. However materials need to be rigid as well. In order to make something light and rigid at the same time you end up with something that "rings like a bell" - i.e. your speakers make their own music which is applied over top of the signal that they are fed.
Often a damped design such as a paper cone or fabric dome tweeter can provide just enough rigidity to be fast(especially when properly supported through appropriate voice coil sizing and cone shape). The advantage of pulp paper and doped fabrics is that they are internally damped and dissipate the "ringing" rapidly...this will make them sound even faster than rigid marterials as you get that sharp transient and nothing more.
A less rigid damped paper woofer or woven fabric cone tends to require a larger diameter voice coil to give it better support. In the case of a fabric tweeter the 25 mm voice coil is sufficient as it couples to the edge of the dome but larger diameter voice coils get expensive in order to achieve a decent Xmax.
Rigid lightweight drivers allow the use a small diameter voice coil on a large woofer with the principle advantage of lower cost, high efficiency, greater bandwidth but inevitably some ringing issues to contend with.