If you use a center drill to spot the hole before drilling, you should be able to avoid your hole "walking" on you. Pics of your jig would be helpful to see if there is anything noticeably wrong with it. Otherwise it's all conjecture. Ask 3 different people to imagine something with a small description and all 3 will have different imaginings of what that something is.