Which is better Coax or Optical cables

thankful - it is system thing. What works for one system might not work for another (even for the same system but in different location). Transitions in optical cable are very slow, therefore it is sensitive to system noise (varying threshold point converts to time jitter) but is not sensitive to ambient electrical noise, ground loops, characteristic impedance mismatch etc.
Transitions in coax are much faster, hence it is less sensitive to system electrical noise, but is sensitive to ambient electrical noise and requires good shielding. It is also sensitive to characteristic impedance matching on both ends while this sensitivity is also a function of source slew rate, cable length and propagation speed thru the cable (dielectric). The best thing is to avoid recommendations and to try as many cables as possible..

Low jitter that you’re looking for means low noise in the music. Jitter induced noise is proportional to signal level and cannot be detected without signal/music. It will manifest itself only as a lack of clarity, harshness, sharp sibilants, poor imaging etc.

@thankful If this is your setup as you listed it:

Oppo 105D
Primaluna Dialogue Premium Integrated
Emerald Physics KC II Pro speakers

I don’t see how you would need a digital cable at all, your integrated doesn’t have a built in Dac nor do you mention an external  Dac. So you just need analog RCA cables to hook your Oppo up to your Primaluna.

As @kijanki already pointed out, it's system dependent. One benefit of optical is the inherent electrical isolation. Lifatec makes excellent glass cables capable of handling 24/192 files.

http://www.lifatec.com/toslink2.html

"But" if a person were to place a DAC between the Oppo and integrated there would be a need one.

""But" if a person were to place a DAC between the Oppo and integrated there would be a need one."

True, however it’s not as simple as adding the DAC and cable. More often than not it becomes an exploration to find the right combination that compliments your system. Have you considered other paths for upgrade?