If the only change you have made is installing DIY speaker cables then the problem logically should be with them.
12 AWG should be ok so maybe you have a short circuit. You need to FULLY remove your cables from your system and do an insulation test.
Are the cables true bi-wire? 4 connections at both ends or shotgun 2x4?
To be double sure put the old cables back in to see if the amp still gets hot.
Have you checked the amp's bias settings?