Straight plunging with endmills does not work well and should be avoided when possible. The geometry of an endmill is not correct for drilling holes. If you can use an endmill that is smaller than the desired hole size and spiral plunge it works OK.
Normally I will drill all of the holes first, then change tools and do the milling operations.