You do not mention te primary purpose for the HT system. Movies, Music, TV, Gaming or all of the above?
Here are my thoughts. Everyone who mentioned that you are missing out on the new lossless formats, namely Dolby True HD and DTS Master are correct. These audio formats truly sound amazing, some of the best recordings I have heard in over 35 years in this hobby. And I am not talking movies in 5.1 or 7.1 lossless (although they are great) - I am talking about music, particularly Blu-ray concerts that sound simply spectacular. Without HDMI 1.3 or higher into a processor or AVR that can decode it, you are missing out. Optical or digital coax cannot carry these signals. They can only do lossy formats which are the old Dolby Digital and DTS.
Now if all you are concerned about is TV or even movies (Blu-ray or DVD), IMHO the old lossy formats are just fine. TV and DVD can only do the old lossy codecs anyway. If you have a Blu-ray player, you still have the lossy soundtrack that sounds just fine for a movie. So running an optical conmnection between player or cable box and your current 5.1 Denon processor is all you need. But if you want music and want to take advantage of the new lossless formats, get an HDMI equipped processor or AVR. In fact an AVR with PRE-OUTS connected into your 5 channel Denon amp would be the most cost effective solution. Just make sure the AVR has PRE-OUTS. They all have room correction, some proprietary like Pioneer MCACC and some licensed like Audyssey that is found on Denon and Marantz products. They all do a good job. Good luck.