Zen,
I design circuit boards. I deal with .001 inch every minute of every day. I have 3 points. 1) one wouldn't want that much "slop" in the hole size. Just tightening the arm would cause it to move more than that. 2) I'd bet you be surprised at how much runout the typical homeowners drill press has 3) at that size, the typical drill bit has a pretty large tolerance.
IMHO, to get the hole that precisely located and sized, one would have to use a CNC controlled drill and a very expensive, high tolerance bit.
Not to mention the fact that you are, usually, dealing with a fiberous material such as wood, or worse yet, mdf.
Just my 2 cents.