The issue for any amplifier, headphone or speaker, is whether the (-) side is active or grounded.
99% of all amplifiers out there only drive the (+) terminal and leave the (-) attached to the signal ground. Sometimes those amps can be bridged so that you use the (+) of channel A for the red and the (+) of channel B as the black wire.
In the case of grounded negative terminals, there’s no difference between the two speaker (-) terminals and they can be safely jumpered together. The (+) terminal however cannot be jumpered together. Even small differences in voltage can cause massive current flows.
The problem is when the (-), like the (+) side is driven, and the same problem applies. Even a small (0.5V) voltage difference can cause very large current flows to occur. A simple 4 conductor to 3 conductor adapter likely shorts the two (-) together which would be bad.