Have you ever tried using the MDI (Manual Data Input) function for machining? It's a lot like manual machining, but instead of cranking, you simply enter the endpoint of each move, push a button, and it goes there. It's good for jobs that don't really call for a program, and it's more accurate than hand-cranking, since you don't overshoot. Plus, you can tell it to move diagonally, or in an arc, and it will do that much better than even an etch-a-sketch master.
I would advise against tearing off the stepper system you've put together, and to learn to use it instead. Or sell the whole thing, mill and all, and buy yourself a manual mill, if that's what you really like to use.