Just my 2cents worth, but I've gone down the road the OP has as I got back into audio because my passion with music was rekindled after setting up a decent HT system.
The OP says he is mainly interested in 2-channel audio and only 'dabbles' with HT. I think his current AVR is holding back what his Aperions are capable of, imho.
I think the OP should consider a different AVR that has the features for outboard add-ons for a future upgrade path; ie preouts.
If the OP is happy with the Yamaha sound, go with something like the Aventage RX-A50 to start. Then in increments, consider adding an external amp(s) with the remaining budget.
I'd look for a good used amp here as that is low risk and best bang for the buck. Really good amplification can provide startling improvements in overall SQ.
I ended up building a hybrid HT/2-channel system coupling my AVR for HT/movies with a nice integrated that has HT Bypass for 2-channel that resulted in a significant step up in SQ for music.