How deep are you going in one pass?
You might try retracting from the hole just to make sure the cutter has cleared its flutes properly. Then, move a few thousandths in the X direction and plunge again, then feed. Us professionals would likely use some kind of a zigzag or helical entry to make the plunge opening larger than the cutter to start with.
You might also ramp down for the full length of the cut, then come back at full depth, fixed depth.