In general I would recommend slowing the spindle way down and starting over. At high speeds the spindle motor is producing far less torque and the belt has far less capture on the spindle pulley. You could be intermittently slipping the belt which would cause all sorts of issues. Keep in mind that calculating feeds/speeds in not an exact science. Machine rigidity, available power, and bit geometry all play a huge factor.
Take a look at:
as I recall the spindle was about 5000 RPM.
The gib adjustment is done by feel. First take the stepper motors off so you can turn the screws by hand. You need to adjust the gibs until you start feeling a slight amount of drag. At this point you should also not feel any 'wiggle' to the table. Tightening them past this point will not help.