Problem connecting Rega Dac to Macbook 2008

Your Macbook might not have Toslink out (Macbook Air 2008 doesn't).  You can verify by checking "About this Mac" and then "Audio".  If it doesn't show S/Pdif there then you don't have it.   You could use USB but it will be limited to 16-bit and 48kHz by your DAC.  It might also be synchronous USB, meaning that the clock for D/A converter is synchronized with computer (asynchronous has independent clock).  You can also get asyc USB to S/Pdif converter.
https://apple.stackexchange.com/questions/46605/how-do-i-tell-if-my-mac-supports-mini-toslink

Yes it supports optical out, the combined 3.5 mm jack. I can see the red light as well, but why can't I get 192khz in the sound/midi settings, the highest option is 96khz, 24bit 2-channel stereo and I don't find the Rega Dac as an option..

Does the toslink only work oneway, and USB should detect the Dac in the list?

My Mac Mini S/Pdif is IN/OUT.  Below copied from "About This Mac":

S/PDIF Optical Digital Audio Output:

Connection: Combination Output

S/PDIF Optical Digital Audio Input: 

Connection: Combination Input

The issue could very well be El Capitan, notorious for usb audio problems and still unfixed by Apple in Sierra. Google "forum el capitan audio problems" and you'll find a ton of unhappy customers. A couple of weeks ago I bought a new laptop running El Capitan and it doesn't allow me to control the volume on my USB DAC, even though it worked perfectly in Mavericks. I downloaded the app "Sound Control" that provides a partial fix but still can't run Audirvana in Integer Mode, so it's basically useless for me. The only hifi way to fix it is to go back to an earlier version of OSX which is not for the faint-hearted. My current workaround is to use my old laptop running Mavericks which I was planning to sell. 

Incidentally most people who have tried all the interface options think USB and SPDIF provide better sound than optical, despite optical being better in theory. Also if you're new to computer audio stay away from iTunes, it sounds terrible for various reasons. You can download a demo of Audirvana for free and compare the two; it's night and day, not even close. (You have to turn on Integer Mode 1 in Preferences). 

Ok but SHOULD the MacBook detect the Rega Dac through the toslink cable or is it only through USB connection? I had an Ifi microdac a few years back, connected with USB and I could find it in the settings

Every new OSX build is different. Can you check it with a friend's laptop using a previous build, Yosemite or Mavericks etc? That will tell you if El Capitan is the problem.

I will try that, thanks!