Speakers move when input voltage applied to the coil creates a magnetic field that attracts it towards, or repels it away from, the fixed magnet. Moving the cone with your finger does the reverse, creating a voltage in the output. This is exactly the way a microphone works.
Shorting the terminals creates a situation where the same voltage being generated by moving the cone with your finger, is simultaneously putting voltage right back into the coil resisting this same motion. As a result it moves hardly at all.
If the speaker is sealed design then you have pretty much turned it into a closed box. But if it is ported then you have created a speaker sized Helmholtz resonator.