If you put in a small value here (0.2mm for example), you will still get the safe Z move but it will rapid back down to the fast plunge height distance above the next toolpath start point. This should speed things up a lot.
There are ways to avoid this move, but an improved stock model is planned which will allow CamBam to move more directly to other points while avoiding uncut stock.
One way to avoid this move is to use a constant plunge spiral lead in. I'll describe this one in your lead in post.