Is this on a new mill? Have you measured and corrected for the actual ball screw travels? If an older mill, you may have some slop in the nuts, bearings, nut mounts or some other place.
I never rely on the travels to give me an accurately round hole, but get close then use my boring head. I just noticed a few days ago my X and Y travels are off, so I got to measure and correct them.
Bob