I used to listen with grills off. Now I prefer the sound with the grills on. With the grills on I hear the music. The speaker becomes more seamless as if the speaker grill is the first stage of room treatment.
Sure, you might hear more detail in some area of the sound without the grills, but it be at a price.