My guess would be a loose coupling on the ballscrew or motor. Anything else would likely result in a "shift" where the position was lost. It looks like what would happen if a coupler slipped a little in one direction, then slipped back. Check all of your set screws and make sure they are tight. If it's something else, it's much more difficult to troubleshoot.
Only the display is rounded. Mach3 does not round the actual position that it keeps internally.I use Mach 3, and have noticed that the fourth decimal digit jumps in 5 digit increments. ..ie although the G code digit is 0-9, the DRO will only show digits 5 units apart, like 0 and 5, and later during the same run 1 and 6 , 2 and 7 etc. I suspect that when I get a round down pass between two round up passes, the groove results.
If your steps per unit result in accurate distances traveled, then you don't want to change it.