I too am fairly new to homemade cnc. I have a purchased kit that total works, but I purchased this kit from ebay,
3 Axis Nema 23 stepper motor 287 oz.in CNC Router NEW on eBay.ca (item 280612051870 end time 06-Mar-11 00:51:38 EST)
the board number is, 3 Axis Driver Board TB6560 3.5A, however I believe in the instructions that came with it, it has TB6560M, not too sure if it is the same board as yours. I too am having trouble, but I can't even get the motors to turn continuous during jog mode in continuos mode. If I select step mode the jog works for the step I set. If I run a file, the motors don't really spin, but the numbers in Mach change as if they spin.
Was wondering if this is the same controler you have, and if so would you be willing to send the XML file for Mach. This is the configuartion file that saves all the settings,