You are right...being sloppy. If all you are doing to convert an SE signal to XLR is adding an "inverting amplifier" you will not get all the benefits of a device being "balanced" and depending on the quality of that section of the amp/premap/DAC, etc... you may get degradation in quality. This is where you can get the phenomenon of RCAs sounding better than XLRs in non-balanced equipment.
If the entire circuit is fully balanced, then it will be cleaner sounding but if the components that balanced signal is being sent to are not balanced then all that excellence is lost.
In practice, this shows up in DACs. Take the Rockna Wavedream SE vs. XLR. In a fully balanced system, the XLR DAC will outperform the SE. In an SE system, the XLR and SE will sound the same.