Can a DAC help?

Your best bet for better sound would be to save that 500 and see if you can get a little more with it and buy an external amplifier, that is where you would probably see the best performance increase per dollar.
Doesnt the 3801 have the burr brown dacs?

A quality DAC will make an audible improvement over the stock Denon DACs. I used an Audio Note DAC 1 Signature and ABed it against the Denon DAC and it was no contest. The Denon phono stage is pretty pedestrian, too.

A DAC goes between a CD player and the preamp or receiver, and takes the place of the internal DAC in the CD player.
If you intend to use the DVD for music then fine, a DAC may or may not have a positive influence - see if you can borrow one first, and AB it using two inputs in the receiver. It all depends on what the quality of the DAC in your CD player is.
If using as a DVD (ie multichannel), then you're SOL, as all the DAC's are two channel.
No, it will not work for 5.1, 7.1, 10.1 or whatever, as the D to A decoding is done in the DVD player.
First step - try both the digital (coax or optical) outputs from the DVD to Denon, and the analog outputs. Rig them up so you can go back and forth, then you'll have a real-time comparison.