Dweller is right -- to expand on that just a little, I've always noticed the distortion on the edges/corners of the picture. The color may be saturated or the picture distorted. Even small distance separations can make a big difference since the strength of the magnetism increases on a squared basis with distance.
BTW, I presented the aluminum foil idea as a hypothesis above. More knowledgeable members shot that one down, so I wouldn't encourage anyone to follow that route. To reassure yourself, just put some aluminum foil on the outside of the speaker box -- surface closest to the TV. If there is no change, you've only wasted a few seconds of your time.