I'm sure you already know this, but the presence of a ground connection indicates no double-insulation, so removing the ground does expose you to the small but real risk of electric shock if the casing ever became live.
Is the improvement in sound worth such a risk?