May I know how does these affect the sound if not "connected" as said?
If you get the pin-2-hot/pin-3-hot connection schemes switched around, then the phase will be reversed . . . refer to all the classic arguments for absolute phase as to whether or not this matters.
If you have a transformer-coupled or "active-floating" output and you don't ground the low-side (usually pin 3), you'll know it . . . you'll get very little sound.
But if you're grounding one side of an "active balanced" (balanced ground-referenced output), everything will still work, but the output driver circuit for the grounded (unused) half will be operating permanently in an overload condition. If it's designed correctly it won't be damaged, but this still isn't good. If feedback for the entire output stage is taken in a balanced, symmetrical fashion, then you may get a very obvious, horrendous clipping. But at the very least, distortion harmonics and half-wave currents from the overloaded half of the output stage are likely to flow through ground traces and bypass capacitors that also serve the side of the circuit that you ARE using, degrading its distortion performance.