So Terry,
After having watched it, I'd say that if you have enough Z travel, I'd highly recommend that you shorten up your collet grip on that tool. It'll be a lot less buzzy if you do. Maybe get ahold of a ball end mill, without the hugely undercut shank, too, which will help with the stability of the tool's tip.
Whats the difficulty you're having with drilling the holes?