When is optical or coax cable or usb best?

You never really know until you listen. Every DAC is different and the designer may have a preference for the type of user he has in mind. Sometimes it is a price point issue.

My listening bias tends to rank this way:
1) SPDIF/Coax 75 Ohm
2) Toslink Glass Fiber Optic Cable (AT&T or similar).
3) USB if the DAC has an Asynchronus USB interface.
*(some are better implemented than others)

Yes, I have listened to all three.
No, I have not listened to every possible cable combination and neither has anyone else. What you will receive are opinions and recommendations based upon limited experience.

Trust your own ears and Good Luck!

After reading so much about using USB out I tested this over the weekend. My preference was optical over USB. The difference was noticeable to my ear. Source is mac mini to a PS Audio DLIII dac. The USB was run straight in to the dac while the optical is fed to a Monarchy Audio Super DIP with coax out to the dac. Before listening I optimized the mac's digital out for each cable.
I have been using the optical out for about a year now and am happy with it.

Thanks. I'm learning that if the DAC is USB you need a very good cable.
And if you're going optical you want glass and not plastic.