Most stepper drives settings go steps to turn the stepper motor 1.8 degrees. So I think you need a setting of 4.
I haven't done the math, but I am wondering if you have not converted from mm to inches. At 4 pulses/step you need 800 pulses/rev (360/1.8=200) to move 8mm linearly,(about .315") to move 12" would take 38.1 Revolutions. (12/.315) to move an inch you need about 2539.68 steps (1"/.315" x 800 step/rev ).
100 steps per millimiter is the correct number for your setup.
The driver DIP switch setting of 800 means "800 steps per turn". The driver has no way to know what kind of a ballscrew you have, it operates in terms of steps, not millimeters or inches.