Speaker grills?

Should they be removed when listening?

Speaker grills can affect the high frequencies which IME is most noticeable in a reduction of an otherwise excellent sound stage. Some grills/clothes can further dull the sound and/or create distortion from reflections off the grill frame edges. BUT some speakers are designed to be heard thru the grills (or cloth coverings) and they will be fine.

Do what sounds best to you, or works best, i.e. keeps the little fingers off the speaker's drivers.
Keep in mind that some speakers are designed to actually be more neutral with their grills on. Harbeths are a good example.

That doesn't mean some might not like them better with the grill off. That's somewhat the same effect as tweaking the treble control up a bit (back when most equipment still had tone controls.) The key is to remember you're the only one who needs to be happy with your choices.
I've seen at least one speaker company indicate on their website that the speaker were designed for the speaker grills to be on. I thought it was Paradigm, but I didn't find it in my very quick reference search prior to this post.

It's personal preference, think of it as a built-in tweak.

I preferred the grills to be off on my previous Inifinity IL-50 speakers, but much prefer the sound of my Focal 836v speakers with the grills on (Note: the tweeter is above the cloth grill).

When I went to the dealer to demo the Focal 816v vs. the 836v the salesman had them setup with the grills off. It only took one song before I insisted that we put them back on. The speaker was brand new and really sounded harsh with the grills off. The speakers sound much better after being broken in, but more than once I've pulled the grill covers and always put them back on fairly quickly.

Personal preference at best.
My Triad speakers were designed to operate specifically with the grilles installed as were my ESP Bodhrans which do not even permit the user to remove the fabric. I've also used my N802's with grilles installed, but in this case it is simply personal preference.
I listen with the grills on because the designer of the speaker, Richard Vandersteen, told me not to take them off. You should ask the manufacturer which way is preferred in your particular case.

Many dealers like to take the grills off mainly because speakers tend to sound a little more lively without grills which might give a better first impression to potential customers and help make the sale.
I have not heard any of the Electra series speakers, but have always preferred the Chorus speakers with the grill covers installed. I have heard several of the Utopia series speakers and I think the drivers have always been exposed. I don't even know of the Utopia speakers have grills.