rats, the situation you refer to is one where the Bal and SE jacks are internally wired together. A typical example are amplifiers that have both Bal and SE inputs, but to use the SE, you need to install a jumper clip into the XLR jack, usually between ground and the XLR's negative hot pin. This is because the RCA hot is connected internally to the plus hot of the XLR while the RCA neg is connected internally to the neg hot of the XLR. The jumper connects the RCA neg to circuit ground in order to make the SE input work. Only now, you are definitely driving only half of the amp's input circuitry with signal.
So you are right, in that you don't HAVE to have an inverter at the input, but then the relative benefits of using the Bal inputs just increase.
So you are right, in that you don't HAVE to have an inverter at the input, but then the relative benefits of using the Bal inputs just increase.