Your problem is motor linearity. One expects a motor to divide a full-step evenly into equally spaced microsteps when it's driven by a microstepping drive. This evenness depends on the motor having good linearity. What you have is a "tin can" motor meaning it is constructed from stamped metal parts. These motors have terrible linearity; the ratio between the biggest microstep and the smallest microstep can be as bad as 5:1.
Machined motors (NEMA-17 and larger) have much better linearity. It can be under 1.1:1 for the best motors.