Transport to DAC Connection: RCA or XLR?

I would say to ask the dealer, as well as call the manufacturer.

When you are dealing with components that have XLR jacks and are not fully balanced, you probably won't hear a difference either way. Some people do claim that using the rca jacks do sound a little better. The difference, however is very small. I've tried it myself and couldn't hear the difference.

With regards to using both connections, I'm not sure. I think it would depend on how the components are designed. If it were me, I would just try it and see how it sounds. I can't imagine that it would harm anything by testing it. It wouldn't hurt to call the manufacturer and see what they say.

I'm not sure what amp has to do with it. OP asks about digital cable between transport and DAC. XLR cable should be better being differential with much higher levels (2.7Vp-p vs 0.5Vp-p) but it is always system dependent. You need XLR 110ohm cable possibly at least 1.5m long. Some high quality transports swing faster but regular transport might swing in about 30ns. Signal travels back and forth (reflection at the end on impedance boundary) in 1.5m x 2 x 5ns/m = 15ns, just missing midpoint of original transition not affecting it, hence not causing jitter at the threshold. I think that 2m cable might be even safer. That's the only case I know when longer cable is better.

As for fully balanced system I'm not even sure if it is a good idea. It increases cost (for more money you can get better amp) and increases chances of poor Common Mode Rejection since amp's and preamp's halves won't have exactly same phase shifts and signal gains. I use XLR differential audio cable between my DAC and power amp but I don't want fully balanced configuration.