I would go with the 1/2-10. The 3/8-12 will start to whip at a lower speed than the 1/2-10. Depending on how long your screws are, and what kind of travel rates you want, it may not make any difference. Nook Industries has a lot of charts and calculators showing the strength and speed at which the different sizes of screws will whip.
http://www.nookindustries.com/acme/AcmeCharts.cfm
If your steppers are marginal, the 3/8-12 will generate 2 lbs of force for each oz of stepper torque where the 1/2-10 will only give 1.6 lbs of force for each oz of stepper torque(assuming plastic nuts).