For best sound, listen both ways and decide which you like better.
Grills help protect often delicate drivers. If you have delicate drivers,like soft dome tweeters say, and kids or pets or house cleaners or such running about, I'd leave them on and do something else relatively cheap and easy to tweak the sound if really necessary, like tweaking placement, toeing in/out, interconnects, even (gasp) tone controls if available etc.
