I would usually look at backlash if you are not getting round holes. The jaggies not sure, maybe lots of backlash and some sticktion. How rigid is the gantry? How much can you bend it if you have a hand size tool in the spindle and you give it a good yank?