A well designed speaker is compensated for the loss in high end due to the speaker grill cloth. The reason that taking the grills off makes it SEEM better sounding is that it's a tab bit brighter. A few more dB's in the high end. Unfortunately, like BLUE TV's in the Best Buys of the world, it sells more, but not truly more accurate.
If you like brighter, GO FOR IT. I prefer audio as a pursuit of the perfect reproduction of the live, hence I keep them on...