Fumigation

1. Remove drivers and crossover.
2. Take all the precise measurements of a speaker cabinette inside and out
3. Burn the speaker cabinettes either in fireplace(save an electricity) or do some BBQ over the charcoals.
4. Take the measurements to the carpenter and re-create speaker cabinettes.
5. Place back drivers and crossover.

Here is an idea:
Open speakers, drop one moth ball into each. Should kill any warms and each ball will disappear in a week or so.

The last time I had warms in my speakers, I replaced the soft dome tweeters with aluminum ones.

Sometimes, especially with heavy speakers, the woodworking dust is not entirely removed after production. Even with some good cleaning, fine wood dust can settle over time. Your speakers might be OK. But, to be safe, you may wish to call an expert to examine them. Unclejeff and others also have good suggestions.