If you haven't you should. A driver is ideally supposed to act as a perfect piston, and if it is loose and shaking that isn't possible, and yes, it is very audible. I'm sure that there are members here who will say that they check them on aa regular basis.
Two years ago, I picked up a 2-year-old set of floor standing speakers from a man in California. He had a nice system and two other pairs of good speakers. When I got them home and checked the driver screws, each of them turned for at least two revolutions! I Immediately emailed him and told him to check his other speakers. He did and wrote back to thank me.