IMO it depends on the speakers first; and to lesser mesurable effect, the gear driivng them.
I used to have Totem Forests and Arros (even bi-amped) .... taking the speaker grilles off was strongly preferred.
Now with a new premium integrated amp and cdp (REGA Osiris and Rega ISIS valve) married up to my new REGA R9 speakers, leaving the speaker grills on has no effect and you get the dust et al protection to boot.
I used to have Totem Forests and Arros (even bi-amped) .... taking the speaker grilles off was strongly preferred.
Now with a new premium integrated amp and cdp (REGA Osiris and Rega ISIS valve) married up to my new REGA R9 speakers, leaving the speaker grills on has no effect and you get the dust et al protection to boot.