Sometimes Toslink may not handle sample rates higher than 96 kHz. If the connection is iffy at a sample rate of 176.4 or 192 kHz, usually what happens is that the output will sporadically mute because synchronization is being lost between the streamer and the DAC.
SPDIF usually handles up to 192 kHz PCM and DSD64 if using the DSD-over-PCM (DoP) protocol while USB can handle up to 384 kHz PCM / DSD128 (DoP) or even 768 kHz PCM / DSD256 (DoP). Some DACs offer native DSD playback over the USB interface, though the streamer must support the feature with that particular DAC.
So it really boils down to what type of sampling rates you want supported and whether high bitrate DSD is important to you.