Are you suggesting that I remove the collet from the clamp and just leave it on the spindle for prolonged periods of time to "enlarge" the collet?
yes, I would do this for an overnight, and maybe place/remove it a few times by itself (ie not inserted in the clamp). If that doesn't loosen it up, I'd find something slightly larger than the spindle (a bolt, or screwdriver, etc)to enlarge it slightly.