jubee,
With those nice ballscrews you should be able to achieve a .0001 inch repeatability but it will take some work. (and probably some different encoders and a lot of testing)
Please explain how you are checking the accuracy. What do you have as far as electrical test equipment? (for checking the servo loop)
I'm surprised that the accuracy was worse with the 250 line encoders on the motor. (I'm assuming you meant 250 line which = 1000 count per rev after quadrature decode)
Did you retune the servos after changing encoders? I've downloaded the info on the Gecko 320 drives and it seems there is no way to control the integrator on the PID loop.This will make tuning interesting as changing the encoder count will change the integrators response.
High encoder counts may be a problem if we run into a pulse speed limit from Mach 3.
Have you ever done a servo size/inertia analysis on this machine?
How deep do you want to get into this? The math behind servo control can get pretty confusing but I think I can lead you through it if you want to take the time.
On the idea of using linear encoders, I'd just forget about it for now. The hardware you have won't handle it and even with big buck hardware it's a real pain.
I'll be glad to help you out but I'm afraid I can't give you a "just bolt this on" answer.
Maybe someone here has built the exact same combination of parts and can come up with a simple fix.
BTW, congrats on getting as far as you have.
Bob