Some basic computer audio questions

I have a late 2015 iMac and the headphone jack doubles as a mini-optical digital output. Not sure but I would guess it's the same for your Mac since I think all iMacs had that feature during that time period.

For analog output you use a stereo-mini cable but for digital output you use a mini-Toslink to Toslink cable or a Toslink cable with a Toslink to Toslink-mini adapter.

The optical cable can go directly to a DAC with Toslink input. I don't know if that's preferable to USB output but it works extremely well for me. It's not necessary in using the optical output but I have a Wyred 4 Sound Remedy Reclocker between the iMac and a Pro-Ject Pre Box S2 Digital DAC to reduce the jitter that's inherent in most optical outputs. Using the digital output will certainly give you much better audio than the analog output.

thanks for the responses!

When I set up my system a few years ago the USB digital output was considered to have more inherent noise and jitter than an optical or coax output and most DACs didn't have USB input. Most DACs now do have USB input, including mine, but this works so well I haven't bothered to experiment.

Alternatively try a dedicated audio streamer like Bluesound Node 2i and see if that provides audible improvements in your audio system. 
It’s available online and comes with 30 days return policy. 

There are decent, relatively inexpensive DACs out there with both USB and optical inputs, Topping and SMSL are two of the names that are recommended often.