Is it taboo to mod a speaker?

Most speaker mods involve things like crossover component quality upgrades, internal wire upgrades, or binding post upgrades, in addition to various cabinet-deadening strategies such as coatings, adhesives/sealants, or sheet appliques. Some folks go as far as changing, removing, or augmenting the cabinet stuffing materials, but both that, and any driver changes done in isolation (in other words, not as a manufacturer-sourced upgrade in conjunction with adequate testing and concurrent crossover mods as needed), will stand a very good likelihood of changing the sound in unintended - and probably detrimental to accuracy - ways. The only other things commonly done involve various stands or footers. I agree with Phasecorrect: If you want an upgrade here, you're better off looking for different speakers in most cases.

Best done by the original designer of the speaker!Changeing XO's or using up to date parts is a plus.Little tweeks like changeing Binding Post and Internal Wiring can be DIY.

To address the question about changing drivers, this is not commonly done for several reasons. Crossover and bass loading are tailored to the specific drivers as a system. Each driver has its own physical and electrical charactoristics. Sometimes very similar tweeters can be switched out without changing the crossover, but otherwise its not just as easy as swapping one driver for another.

Yes,If you change Driver's you have toi redo the XO's then have the Speaker's tuned again.I have to get mine done after adding a Mid\Bass Driver,but the sound is better than having the stock driver's so I am happy with the results because they are very close to the Stock models as far as the freq.'s crossover points.

Moding speakers is always something(like anything else), that you can try, indeed. You just have to do "one thing at a time", and see what you've got when you do a modification! If it works, you leave it. If it doesn't, you sould probably change it back!
I've rewired speakers, "braced" cabinets", changed drivers, and changed "single post" speaker designs into "dual post/bi-wire/bi-amp" capable speakers...but never change crossovers. The results are sometimes improvements, but some times not!
In general, if you're not a speaker designer per se, you can only cross your fingers. But, you try and see, and then you'll know for sure!
Good luck