If you gave him a verbal with no deposit, then backing out would only discredit you as a person and your word. Ultimately your conscience has to reconcile that. However, you gave him a deposit and your word. That is different. In some ways (and I don't have all the facts) that could be construed as contractual. If there is a "cooling off" rule (one that is clearly stated in writing, or if verbal can be legally enforced) that allows you to change your mind and you exercised it within the appropriate time frame, then I'm not sure what the problem is. He should give your deposit back.
There are two sides to every story and the truth almost always lies in the middle.
There are two sides to every story and the truth almost always lies in the middle.