no expert here but I have a couple ideas for you....
1.) have you check the hole sizes with a bore guage?
every caliper I've ever seen has a slightly flate section on the measuring edges (i.e. they're not sharpened to a knife edge, but have a small flat) ..... which will introduce some error and an ever so slight undersized reading.
2.) how do linear machined features measure on the mill? spot on? or a hair light?
Maybe your calibration is ever so slightly off.
3.) Any back-lash on your screws/nuts.... that would also introduce some error
Sounds like you have a great set up... should be able to get it figured out.