Going from the Denon to the NAD is a sideways step at best. I wouldn't bother. Save your money.
Going from a receiver to a good integrated, or separates, is not a sideways move, at least if you choose a good product. Your own ears told you that when you listened to the Rotel/B&W set up as compared with your current system.
The reason is pretty logical. Mass market receivers are sold to a price point in a very competitive, low-margin market. The manufacturers cut costs by using the cheapest design, parts and build that they can. And they put as much in the box as they can in order to convince consumers that they are getting "more" for the money. However, the "more" is not quality sound. It's lots of flashing lights and buttons and knobs, and "sound enhancing effects". You also get cheaper parts, different components sharing the same power supply jammed together in the same box causing electrical interference with each other.
Now you certainly can do a good job building a receiver or integrated, but it will be way more expensive than the receivers you find at Best Buy or Circuit City. They are sold to a different market segment.
Separates allow the manufacturer to do one thing well. It also makes it easier for audiophiles to upgrade as they can do it one component at a time. The high-end market was always separates.
Now as it turns out, recent years have seen the production of audiophile integrateds and receivers so there are a lot of good products in these categories. But they're going to be more expensive. And you don't find them at the mass market stores.
As Macrojack says in his post, there is always the danger that you can get carried away in the very expensive and never ending search for audio nirvana. However, that's your own decision as to how much you want to spend for what level of improvement you perceive.
You are at the point where a relatively small amount of money will move you from the mass market to at least the lower levels of really good equipment and sound. It's at that point that you have to take a deep breath and decide if the incremental improvements are worth the cost. Diminishing returns will set in but you're not there yet.
Try going back to your dealer and listen to what good integrateds he may have.