How are you putting the holes in? Are you helical milling them with a finish pass? Are you stepping them down with a circular finish pass? Or are you finish boring the holes?
I have learned that I haven't been able to get a hole better than .001 in roundness. If I need it better than that, I'll bore it.