to diagnose, this needs a lot more info.....how far off are things and what's the goal...lining things up by hand? is this a cnc or a drill press? are you spotting the hole first? what exactly is the problem; ie drilling over sized, hole not where its supposed to be, hole isn't straight etc.
first thing to look at with a drilling problem is the drill grind itself, that's assuming basics are covered...its the drill grind that keeps things straight and inline more than how eccentric the chuck is