I don't know much (anything) about your Yamaha, so wouldn't be able to offer any advice on settings. I suspect you've fiddled with bass settings "per mode". I know some AVRs offer the option either a basic crossover to the sub, or summing the low frequencies of both the mains and sub. This latter setting can cause bass bloat, especially if your mains aren't up to the task.
Best of luck with your setup!
Best of luck with your setup!