when the cutter is over the work, there's force involved and with a flycutter this force has some leverage. it is very normal, in fact its a good sign everything is working well to the get cross hatch pattern - ie cutting on both the forward and back 180. with everything right, the tool marks should be about the same.
rigidity of the machine & set up, the depth of cut, feed, speed, dia of flycutter, and the shape of the flycutter tool all play there part. you have to work these things out or tell us what you're doing and we'll help
what is a mil? if its a millimeter, i've found the problem
how did you square the mill when - trammed with a good indicator? how square did you get it?
in general, reduce the cutting force and see what happens - slow it down, light cut, slow feed, hss bit with lots of rake, etc, at the same time tighten everything up; setup, gibs, quill lock etc. flycutting demands a lot from a mill, don't how well the sherline is up the task, but if you're flycutting a 6" slab of CI its a BIG difference than a 1" piece of AL