I would say that overall the Rotel is a better sounding (i.e more neutral, transparent) receiver but the Denon has more processing/HT features. The Rotel handles transients better (attack and decay of instruments, crispness of treble instruments i.e cymbals, bells, percussion, horns,.. has much better dymanics and is better at reproducing the little details, nuances and subtleties of sound that make a performance seem more real. I would also agree with MKfischers response that the Rotel has better amps. Don't let the Rotels 75 watt per channel rating vs. Denon's 105 watt per channel fool you into believing the Denon is more powerful. Those are just spec's. In real usage (at your home or direct demo comparison at the dealer) you will find that the Rotel has much more balls, headroom and will produce louder volumes with less effort than the Denon. Finally, the Rotel 5 year warranty smokes Denon's 1 year warranty.
Now, on to the positive attributes of Denon. The Denon has a very smooth and warm sound (i.e colored)but many people (incl. myself) like it. The sound is also even through all frequencies (bass, mid-bass, mid-range and highs) so it doesn't over-emphasize one particular frequency band. This attribute lends itself well to overly bright speakers by taming the brightness. It also creates a very smooth "wall of sound" for home theater and DVD use. It also has a better Re-EQ feature than the Rotel's (Rotel's is proprietary while Denon uses THX's version..please note, however that the 3803 is not THX certified..neither for that matter is the Rotel). The Video upconversion feature is also nice ( all inputs..i.e composite, S-vide0, etc can be set up to go out component video). The Denon has a very slight edge in DSP procesing and the latest movie formats.
Finally, the Denon has 7 channels of amps whereas the Rotel only has 5 (However, I don't view this as an advantage to Denon since the quality of Rotel's amps far exceeds Denon's and most people don't have rooms in their homes that allow for seven speakers. Additionally, the Rotel has pre-outs for additional channels should you wish to expand to seven).
One great feature both receivers share is an adjustable crossover (40,60,80,100,120) that allows for bass management flexibility should you decide to change your speakers or configuration at a later date. The Rotel also has a second sub-woofer pre-out should you decide to use two sub-woofers instead of one. That is the prevailing new trend in home theater. I forgot to mention that the Rotel has better bass (more slam, weight, tightness) than the Denon. However, Denon's bass response is no slouch..it sounds full and rich..just not as tight with less slam.
Bottom line is the Rotel sounds better, is more transparent and has more power. If music is more important to you (or 50/50 with HT) go with the Rotel. If connectivity, processing features and HT usage is more important go with the Dennon. The 3803 is definitely a good sounding receiver and you would probably be very pleased with it. However, if you are a budding audiophile and want the best of both music and HT go with Rotel. The Denon sounds good, the Rotel sounds better. Neither are great...for that you'd have to step up to separates like the Anthem or EAD Theater Master. But those are in the $3,000-$6,000 range so why bother?
Charles
P.S- The Rotel 1065 sounds significantly better than the 1055. Also, don't even consider Marantz as the Rotel completely blows it away! FYI...I own the Denon 3803 and the Rotel 1065 so I'm talking from experience.