the grills

It depends on the speaker design.

For best sound, listen both ways and decide which you like better.

Grills help protect often delicate drivers. If you have delicate drivers,like soft dome tweeters say, and kids or pets or house cleaners or such running about, I'd leave them on and do something else relatively cheap and easy to tweak the sound if really necessary, like tweaking placement, toeing in/out, interconnects, even (gasp) tone controls if available etc.

My wife and I have 2 "stinking cats" that don't go near my Wilsons.

My dog doesn't go near my stuff either.

Unfortunately, kids and others milling about the house can be harder to train I've found.

Plus, accidents do happen......

I always leave all my grilles on.

The only time my grills are on is when either of my 2 grandsons come over. With my mission/cyrus 782's speakers I couldn't hear any differences with the grills on or off. But I now have Dynaudio's C1's. The soundstage disappears with the grills on and opens up dramatically with them off. Which ever sounds better to you and how important is protecting the drivers are to you.