The following reference is a bit technical, but well worth reading:
http://www.jensen-transformers.com/an/an003.pdfThere is no general answer as to which is better, because many variables are involved, encompassing the equipment design, the noise environment, the ac power distribution scheme, the parameters of the specific cables being used, etc.
Note in Figure 2.1, though, that a properly done rca-to-xlr adapter arrangement WILL provide some degree of common mode noise rejection, although not nearly as much as adapting via a high quality transformer. But if the amp is not "fully balanced," meaning that its internal signal path is unbalanced, that benefit may or may not be outweighed by the sonic effects of the extra stage that would be present at its balanced inputs to convert them to unbalanced internally.
Regards,
-- Al