Is it taboo to mod a speaker?

Everyone else has covered pretty well. The only mods I've done and would recommend to speakers is deadening the cabinets with "dynamat" equivalent. That one is usually an improvement with no downsides/tradeoffs. Changing a driver and doing it properely is quite a bit of work and usually not worth it. I've read conflicting claims about the foam rings to help with diffraction. Someone has done some measurements on them and they don't quite work as the manufacturer claims--I believe.

Most speaker mods involve things like crossover component quality upgrades, internal wire upgrades, or binding post upgrades, in addition to various cabinet-deadening strategies such as coatings, adhesives/sealants, or sheet appliques. Some folks go as far as changing, removing, or augmenting the cabinet stuffing materials, but both that, and any driver changes done in isolation (in other words, not as a manufacturer-sourced upgrade in conjunction with adequate testing and concurrent crossover mods as needed), will stand a very good likelihood of changing the sound in unintended - and probably detrimental to accuracy - ways. The only other things commonly done involve various stands or footers. I agree with Phasecorrect: If you want an upgrade here, you're better off looking for different speakers in most cases.

Best done by the original designer of the speaker!Changeing XO's or using up to date parts is a plus.Little tweeks like changeing Binding Post and Internal Wiring can be DIY.

To address the question about changing drivers, this is not commonly done for several reasons. Crossover and bass loading are tailored to the specific drivers as a system. Each driver has its own physical and electrical charactoristics. Sometimes very similar tweeters can be switched out without changing the crossover, but otherwise its not just as easy as swapping one driver for another.

Yes,If you change Driver's you have toi redo the XO's then have the Speaker's tuned again.I have to get mine done after adding a Mid\Bass Driver,but the sound is better than having the stock driver's so I am happy with the results because they are very close to the Stock models as far as the freq.'s crossover points.