Couple things that may help, no guarantees:
If the shank of your bit is thin and your running the RPM of the spindle high, the bit may succom to centrifugal forces product a tapered hole
I didnt open your code, however if your pocketing your hole in multiple Z increments, a small amount of play in your drive system will be unnotticable in the beginning but at the end of your program the dimensions could be a ways off.
Yeah, it cant be tram. It is an interesting problem for sure.