When I upgraded the drivers in my Alon speakers years ago, Carl Marchisotto recommended I use a certain sized screw that was just bigger than the hole in the driver, which was screwed into the front mounting hole of the driver until it became engaged and then I was able to pull the driver out from the front, similar to what was recommended by @elliottbnewcombjr
I suggest trying the recommended methods posted below before prying against the cabinet, which seems to be asking for trouble.