This sounds like what happens when the Z axis loses steps while trying to lift the head. It's probably doing it a little bit with the other types of cuts and you're not noticing them. But a 3D toolpath requires lots of little Z moves, and they're adding up. Try reducing the acceleration on the Z axis by about half, and see if that fixes the problem. You could also try reducing its velocity. If none of these things helps, you might think about counterweights or gas struts on the Z axis. Also, check the coupler, to make sure it's not coming loose, which can also cause your Z level to drop, and make sure your collet is really tight - spiral-flute tools can gradually pull themselves out if they're not.