Whats the best type of digital cable?

XLR or Optical?
In theory, the XLR balanced configuration is superior. Line noise is rejected just like when using a balanced interconnect. TOSLINK optical is a complete waste of time because the SP/DIF signal is truncated for "consumer" grade TOSLINK. ST Glass optical is great because it carries the full SP/DIF signal *AND* electrically isolates the DAC and Transport. However, in my experience, the balanced digital links have sounded better. However, neither cable will stand up to the new I2S links. (I2S is a 15 pin link which carries audio and clock signals - the best implementation possible is the slaving of the transport to the DAC) Keep a look out for I2S and naturally don't buy balanced, optical or I2S before you listen to them all.
In my experience any good 70 or 100 ohm coax cable, RCA or XLR terminated will sound better, more musical, and a lot less grainy than optical connections. Of course the opposite will be true if you use bad coax cable.
ST optical is the best, but just too expensive. So stick with balanced SP/DIF (XLR) or plain SP/DIF. If you have XLR connectors to both transport and DAC go for it, but this is not a 75 Ohm connection (I think it is 110 Ohm) so it needs a different cable (of course it has three wires). Plain SP/DIF must be coaxial 75 Ohm, so a BNC connector is the best. Most of the machines though have RCA connectors. Serious cable manufacturers have introduced factory-terminated cables with 75 Ohm RCAs, so go for on of those, like the Kimber D-30 (copper) or D-60 (silver).