The best digital audio out of coaxial, or optical?

Toslink, no matter the quality, is no comparison to SPDIF.
The reason is that to create the optical signal it must go through an additional optical EMITTER and an optical RECEIVER in addition to the rest of the circuitry. All things being equal those optical devices mess up the signal.

I use glass optical and coax interchangably and have not noticed any more changes than one would experience by trying two different cables of the same type but made by two different mfg's. These days they are all very good once you get past entry level cheapo stuff. By the length you need and avoid sharp bends and you will be good to go.

Recently I bought a Logitech transporter. I have a oppo 980 connected to it as a transport. I tried a Siltech silver cable (coax) and a Van den Hul optocoupler (toslink)as interconnects. I couln't hear any differences whatsoever between to two of them. The rest of my system consists of Mark levinson 336 and B&W n802, so resolution should be good enough.