Obviously you are losing steps, I would imagine.
Is there any reason to expect that more torque is required to drive the Z down than to raise it? How does everything feel mechanically when you turn the axis by hand?
Are you plunging into the work? What kind of materials do you cut?
Have you run trials where you simply run the Z axis up and down in the air for a while? Does it lose position?
Have you tried homing the machine after every Z plunge?
Sorry, I can only ask questions That dilemma is why I would never run open loop feedback on a machine tool.