Note that Harbeth speakers are designed to have the grills on. Most speakers designed to have the grills on are more accurate and smoother with the grills.
Audiophiles are conditioned to search for ever more "transparency and detail" so "why would I want grill covers between my ears and the drivers?"
They take off the grills, now the frequency response changes a bit, usually a bit of a peak in the higher frequencies and then it's "wow, listen to that new air and detail!"
But I find this is almost always at the expense of smoothness, neutrality and coherency. It's a more "look at me" from the upper frequencies/tweeter" sound. But...if you like it better you like it better.