What CAM software are you using?
I use MecSoft CAM and have had a similar issue on my cnc machine. I checked every on my machine. Turns out that the CAM software was causing the drifting in the cutting process. I re-computed the gCode and it took care of my problem. I want to say it was some sort of numerical rounding issue in the math. Not sure, but it was just weird in that my drifting only took place on two letters of A and S in the wording of allspark.
All the other letters had no drifting taking place. Which pointed the problem away from the actual machine hardware and pushed the issue some place in the software computational issues.
My fix to my drift was re computing the gCode. And I was using ARC's in my cutting process.