Strange to me it seems like your machine might be deflecting? I take a shallow skim cut sometimes and it works fine.
But if you can’t do the shallow cuts just do some test parts until you hit it dead on the first time. I have a product I do that have bronze bushings presses in and I interpolate the hole then give one finish cut and it’s done light press fit no fiddleing.