I am a little puzzled by the ratio you are using as 50800 would be typical for a metric screw. Do you have a mechanical reduction in there as well? Is this a backlash free reduction?
Are the thrust bearings on the ballscrew in perfect adjustment? Encoder couplings tight?
Have you tried a trial cut at two different diameters that span the range of swing of the machine? Does it cut those two diameters correctly.
What about heat? Could this be a heat related issue where the spindle housing is warming up?
While I would not expect the readouts of the display to demonstrate a mechanical error, a series of programmed moves could be used back and forth between two positions, and a dial indicator to check for migration of the endpoint.
Have you tried adding in a return to home command at the end of every run of the program?
Do you have a shielded encoder cable? Is the shield grounded at only one end? Is it running in its own conduit?