The best digital audio out of coaxial, or optical?

Coax is generally superior to the plastic Toslink cables. Those that use a Quartz or Glass conductor can be very good, however they tend to be much more expensive.

One nice thing about optical is that it does not link the grounds of the two components like a metal cable does, which can help resolve ground loops and even minor sonic issues.

AES/EBU properly implemented tends to be better than either.

Toslink, no matter the quality, is no comparison to SPDIF.
The reason is that to create the optical signal it must go through an additional optical EMITTER and an optical RECEIVER in addition to the rest of the circuitry. All things being equal those optical devices mess up the signal.

I use glass optical and coax interchangably and have not noticed any more changes than one would experience by trying two different cables of the same type but made by two different mfg's. These days they are all very good once you get past entry level cheapo stuff. By the length you need and avoid sharp bends and you will be good to go.

Recently I bought a Logitech transporter. I have a oppo 980 connected to it as a transport. I tried a Siltech silver cable (coax) and a Van den Hul optocoupler (toslink)as interconnects. I couln't hear any differences whatsoever between to two of them. The rest of my system consists of Mark levinson 336 and B&W n802, so resolution should be good enough.