Theres alot than can cause this.
With such a small drill, and ar over 6 1/2 times diameter, it's going to lead off easily.
I break up problems into 5 areas;
Should go without saying, but surprising how many times little things in the code get missed.
First, is the drill sharp, and is it sharpened correctly?
Speeds & feeds, tto much/too little tool pressure? Whats the finish like?
Chuck pressure, are you deforming the tube?
Is your tool holder holding the tool parrallel to the bore?
I would be tempted to ream rather than drill.
I don't know if core drills come that small, but that would be better than a 2 flute.
Is the turret true to the chuck? Centered and square.
Is your Axis straight, meaning does it feed straight in Z? Do you normally make taper edits when turning or boring?
Not always in that order, but generally a good start.