I'm probably gonna take some heat from other "goners" but I recommend (and so does Home Theater Magazine) using the coax for the CD and the toslink for the DVD.
Also, hook up the analog outputs of the CD player and compare the sound quality of the DAC's in the Denon DCM-460 to the ones in the AVR-3802. You might like the analog outputs of the DCM-460 better which may render the other question moot. And the volume control on the DCM-460 remote doesn't work when you are using digital outputs.
My experience is that the toslink vs. coaxial doesn't make much difference on Dolby Digital or DTS, but it makes a noticable difference on Red Book audio CD.
Also, hook up the analog outputs of the CD player and compare the sound quality of the DAC's in the Denon DCM-460 to the ones in the AVR-3802. You might like the analog outputs of the DCM-460 better which may render the other question moot. And the volume control on the DCM-460 remote doesn't work when you are using digital outputs.
My experience is that the toslink vs. coaxial doesn't make much difference on Dolby Digital or DTS, but it makes a noticable difference on Red Book audio CD.