I have noticed a problem with the way the signal to control the speed is processed.
If you run the machine in manual, and keep turning up the spindle speed does it get to a point where it stops if you increase the knob too far?
If that's the case try reducing the commanded speed to 3000 RPM, and sneak it up a bit at a time (mach3 slider on second screen) until you find the maximum that will work. The bottom end of the speed range may have problems too.
Now backoff 10% and use that as your maximum program speed.
(IMHO) The AD conversion algorithm in machine control, is poorly designed.