Perhaps the domain spins will return to their quiescent state, and perhaps not. It would require a careful assessment of the relationship between the fixed magnetic field and the signal-carrying conductor to see if this is the case. I would suspect that some work on this has been done by speaker manufacturers to determine the shielding necessary to prevent any sort of interaction. Also, magnetic materials are tricky since they have residual flux whose value changes depending on the last circuit operating point - even if it is simply induced. This problem becomes very significant in large transformers where you can have a huge power surge when you bring one online if the residual flux level is high. Whether a fixed speaker magnet can have this problem enough to affect the sound is another matter...I suspect the woofer surround has a much larger impact than that at any rate.
Either way though, there is more to it than the speaker and there is no clear answer to the question! Fun to think about though.
Either way though, there is more to it than the speaker and there is no clear answer to the question! Fun to think about though.