1) Start with a site like this or this.
2) Read up on DC, AC, Resistors, Capacitors, Inductors, Semiconductors (mainly Diodes & Transistors)
3) Get a copy of the Loudspeaker Design Cookbook (if you want to know about speakers too)
4) Search up Rod Elliott and Douglas Self and maybe buy a book or two
5) If you want some help understanding get an account with diyaudio.com.
6) If you’re still interested download some simulation software, which is a great way to learn (TinaTI / Spice / speaker design software)
7) Design and make some stuff
8) Go to diyaudio.com to find out why the stuff you just made didn’t work
9) Read up some more
10) Make some better stuff
11) Goto 7
Beware... once you reach point 7 you are in an infinite loop that cannot be escaped!
edit: meant to say... don't get too hung up on the maths if it's not your thing you should still be able to understand the principles.