You say ''lower to the switch'' Normally you contact the home switch going up. That way gravity is working for you by pulling down on the axis. Going up helps overcome any backlash and stiction. I would not rapid off of the switch, maybe come off at 5 IPM then reverse direction at about 1 IPM for the final move.
With a +/- 3 micron switch, you should be able to hit +/- 3 microns.