Is it possible that your spindle and tooling is not "trammed" to your table surface? Maybe the problem areas have the cutter tip going too deep into the circuit board substrate.
Does the degradation of the milling show up in a specific area (or areas) of the board? Is it possible that, rather than being related to the start and end of the job, instead the problem is related to the location on the board that is being milled? Is it possible that your board material is warped in some areas and is not sitting flat on the table?