Fumigation

One way to see if you have something exotic in them is to seal then in a heavy-duty plastic bag. Leave it be for a while and then see what kind of dust might be getting thrown out by their activity. Also, sealing them in plastic will prevent further invasion to your home.

I would question your assumptions that there is something living in there. If you do seal them in plastic wrap some tissue paper around them at the base as whatever is coming out will be easier to see.

I would consult an expert keeping in mind that most pest control inspectors are only familiar with local bugs.

1. Remove drivers and crossover.
2. Take all the precise measurements of a speaker cabinette inside and out
3. Burn the speaker cabinettes either in fireplace(save an electricity) or do some BBQ over the charcoals.
4. Take the measurements to the carpenter and re-create speaker cabinettes.
5. Place back drivers and crossover.

Here is an idea:
Open speakers, drop one moth ball into each. Should kill any warms and each ball will disappear in a week or so.

The last time I had warms in my speakers, I replaced the soft dome tweeters with aluminum ones.

Sometimes, especially with heavy speakers, the woodworking dust is not entirely removed after production. Even with some good cleaning, fine wood dust can settle over time. Your speakers might be OK. But, to be safe, you may wish to call an expert to examine them. Unclejeff and others also have good suggestions.