Is it taboo to mod a speaker?

Best done by the original designer of the speaker!Changeing XO's or using up to date parts is a plus.Little tweeks like changeing Binding Post and Internal Wiring can be DIY.

To address the question about changing drivers, this is not commonly done for several reasons. Crossover and bass loading are tailored to the specific drivers as a system. Each driver has its own physical and electrical charactoristics. Sometimes very similar tweeters can be switched out without changing the crossover, but otherwise its not just as easy as swapping one driver for another.

Yes,If you change Driver's you have toi redo the XO's then have the Speaker's tuned again.I have to get mine done after adding a Mid\Bass Driver,but the sound is better than having the stock driver's so I am happy with the results because they are very close to the Stock models as far as the freq.'s crossover points.

Moding speakers is always something(like anything else), that you can try, indeed. You just have to do "one thing at a time", and see what you've got when you do a modification! If it works, you leave it. If it doesn't, you sould probably change it back!
I've rewired speakers, "braced" cabinets", changed drivers, and changed "single post" speaker designs into "dual post/bi-wire/bi-amp" capable speakers...but never change crossovers. The results are sometimes improvements, but some times not!
In general, if you're not a speaker designer per se, you can only cross your fingers. But, you try and see, and then you'll know for sure!
Good luck

I have bought some audioquest cable and replaced wire.I think the designer did his or her best and is best left alone.It reminds me of wisdom . if you are looking to change something,you may well find it somewhere else and sell your speakers unmodified for a good price