Which is better Coax or Optical cables

Plastic fiber TOSLink cables -- basically everything available on Amazon -- are junk. Cheap plastic cables can barely handle the bandwidth required for 44.1/16 (for the most part not even that) forget about anything higher. 

The problem stems from losses due to internal reflections. You need very fine, high-purity quartz-class conductor bundles and polished connectors to reduce reflection losses to to negligible level. Of course the quality of the receiver and transmitters play into this.

There are also high grade plastic cables out there such as the mid-level AQ and Supra cables that are supposedly the sweet spot for price-performance. I haven't compared any glass cables, but I do have a Supra cable that cost around $80, and have confirmed it competes in SQ with a filtered/isolated USB connection running off my dedicated audio PC. 

Check the spec's on toslink vs coax connections.  Toslink is limited to 96khz plus it's construction is generally inferior to coax, certainly when compared to a really well designed coax.  I use an MIT Magnum Digital Coax which produces superior sound to my previous $12k balanced cable running analog from my Krell Cipher (which I now sold).

For short length less than 15ft and with 24/96pcm, toslink is just as good as good coax. With toslink, There's no electrical comnection between components for those who care.  Plastic vs glass for such a short distance is silly... There's no loss of 0's and 1's with plastic.
Coax can stream beyond 24/96pcm.
Both can't do dsd.  Only coax can stream dsd converted into pcm using dop stansard (24/176.8)...
so i guess coax is better, it has less limitations.
No need to spend a fortune.  Just good shielding and 75ohms connectors.

Hands down without a doubt Coax or RCA is way better then Toslink/Optical for handling digital signals. the list in order is XLR,Coax,Optical. Toslink is last because it adds the most jitter of all. (jitter is digital distortion)