How long does it take a decent quality speaker to "wear out"?

Durability and lifetime depends on specific design.  I have heard speakers with 70 year old drivers that sound terrific.  Paper woofers with pleated fabric surrounds can be quite durable compared to even modern woofers with rubber surrounds.  My midrange compression drivers are about 80 years old. Even speakers with surrounds that go bad can be long-lived if you accept that surrounds have to be replaced every twenty years or so.

i own a set of B&W 801 series 2 with Kevlar cones which were never played very hard never in direct sun light which i purchased way back in 1984 all i have done

is to keep the cabinets oiled so they would not dry out and the dust off the covers with the original brush i received with the speakers, I am using a Krell built

Bass alignment Filter which was designed for that series along with bi-amping and they still sound as good as the day i purchased them.

Still my Dunlavy’s (bought new) show no signs of age but they are well looked after.

Crossovers are still like new, no electrolytics to worry about, no corrosion or bad joints, internal cabling perfect. Never pushed really hard, but always connected to big amplifiers, rubber surrounds talc powdered every 3 years and cabinets always clean and shiny as new. Keep them or not they should perform nicely for years to come. Still if there is any deterioration in sound due to age i cannot tell but i am not going to replace any part just to find out.

Have an old pair of Spectrum 208A made in Toledo,Ohio.  The firm closed in the 90s.  Recently had them re-grilled and re-foamed by a speaker guru outside St. Louis.  

Still have the same cross overs ... still surprise me!  

Until a few years ago was listening to a 40 year old pair of Vandersteens, then traded them for a 30 year old pair of Vandersteens.

No longer own them either.

Both sounded and performed as new.

Who knows when they might "wear out"?