Has absolutely nothing to do with Deskcnc, G-code is G-code the world around. I would first suggest that you make sure your trim pots on the 540 are properly adjusted
From the 540 manual~
STEP 7: ADJUST TRIMPOT
The G540 is composed of four individual G250 motor controls, each of which has a trimpot on board to adjust low speed smoothness. In your CNC program set the axis being adjusted to move at 2 revolutions per second and use a non-conductive screwdriver to turn the trimpot in the ADJUST hole on the top of the drive. If you do not have a non-conductive screwdriver you may use a standard screwdriver, but be careful not to touch any components other than the trimpot. The trimpot has 270 degrees of motion and will have an audible “sweet spot” when it is in proper position. This position will change with each motor so be sure to readjust if you change motors.
basically run the motors at 2 RPM or so and adjust each trim pot until you will hear the sweet spot.
I would also look into dampers on your steppper motors, just a flywheel helps, most steppers in copiers and scanners have dampers installed, this is the main reason you can buy steppers with dual shafts, the outside shaft is more for a damper than a manual knob or dial.
I run Gecko 251's and even though they have mid band resonance dampening built in I run dampers on my motors to help with vibrations at low feed rates.
Its wierd that when you manually jog its fine and only does it while running code, Mach 3 has way too many setup options, you may find some advice in the mach forums.
how many steps per inch are you running? having a gear or belt reduction so the motors are at a higher RPM might be beneficial on a hot wire cutter where you dont need insane rapid or cutting speeds. N/m I found your at 600 steps per inch, Which is pretty low but not insane, I think upping to 2000-4000 might help, with more than just vibrations.... 600 steps on my last machine was only .075" and it could rapid at 400 IPM....
What you basically built is a stringed instrument, any vibrations from the machine will transmit to the wire, What you need to do is try to design the machine around the resonant frequency of that wire, so any vibrations produced by the machine are well outside of that resonant range. Changing the resolution is probably the best method, and adjusting your wire tension is probably the easiest. You also have to factor in the resonance of the belts, might try adjusting tension there too.
Most of the current state of the art belt drive setups dont have a tensioned belt hanging in the air for that reason, the drive belt is meshed with a stationary belt along one of the guide rails and loops up and over the drive pulley running on idler bearings, this way theres a few inches of belt free to vibrate, even if the axis is 100 feet long.