If I recall correctly, some of the other guys were remarking that their stepper motors lost a lot of power when they decreased the step angle. This will cause you to loose steps. This is also the problem with an openloop system (no feedback): the control has no way of knowing that the machine has not reached position because it is just pushing out so and so many steps.
Try going back to full step mode and reduce your accel/decel and top speed settings to see if you can make it work at low speeds. Then, you can work your way back up to where it begins to misposition. This is something you can try at least until the more experienced stepper guys can post an anwer here for you
