The more feedback you have, the less the speaker can affect the amp.
I think it is worthwhile to get very specific. The more feedback the less the speaker’s impedance can affect the amplifier’s frequency response. That is, it behaves closer to an ideal voltage source. OK, that part I agree with, but this question is a little different.
The more feedback you have, the more the speaker’s movement is likely to create the amplifier to respond.
Consider a situation with two amplifiers, their inputs shorted, while music is played in the room from a completely independent source. Maybe even a drum set in the room.
One amplifier has zero feedback. In this case the speaker’s microphonics will not have any effect on the current flowing in a feedback circuit, as it does not exist. The voltage at the outputs may rise and fall but there's no current.
The other amplifier has some feedback. In this case the speaker’s micophonics will excite the feedback circuit. Now we have a situation where currents are flowing in the amplifier that have nothing to do with the input signal. In this case, the output voltages are forced to zero by the active response of the amplifier.