It sounds like a combination of backlash and tool deflection. Have you tried measuring the backlash (the distance traveled by the motor before engaging the screw when it changes direction)? How long is your tool and what diameter is it? The longer and thinner the tool you use, the more it will tend to deflect when pushed against the material, and the harder the material is, the worse it can be.