Try to run the motor with the Y axis cable to see if the motor faults out. Also, take a look at the motor setup again. Can you run the motor from the setup program with the USB cable?
If you made the signal cable, is it possible that the step and direction inputs are inverted? Input A is direction and input B is step as I recall.
Are the returns on both of the inputs properly connected?