Easy answer: whichever sounds best to you. In my experience, most sound a little more transparent and open with the grills off, especially if the grill is perforated sheet metal. But some speakers a resonically engineered for the grills to be in place, in which case removing the grills *might* make the speakers sound a little bright.
In the case of my Maggies with permanently attached grills plus that I'm deliriously happy with the sound makes it a very convenient non-issue with me.
