That is, the resistance to my fingers or external sources is causing the amplifier to respond with enough current to keep the voltage at zero, so it's not a zero sum game, I think.
@erik_squires You exert far more energy with your fingers on the cone than sounds in the room will! To me this seems a non-issue. Again, if the amp is on, the speaker is harder to move due to the output impedance and feedback having nothing to do with it. To show this for yourself, put a 0.5 Ohm resistor across the speaker terminals and you'll see what I mean. The resistor has no feedback yet the speaker is much harder to move.
When I'm at audio shows and there are multiple speakers in the room, I short out the terminals of the unused speakers so they won't resonate.