The CGS high helix should do that cut like it was nothing so too much material for the tool isn't the issue , too much material for the machine is more likely . Sometimes conventional milling will solve the problem and CGS does recommend that any cuts below 1 1/2 dia be cut conventional , I know that your cuts aren't to that depth but my point is that its worth trying on a less than rigid machine . .005 chipload/tooth and 8000 rpm is the start point for those end mills , since yours will only run at 4000 then you should be running at max and I'd start at .006 chipload , if the chatter persists then youve got other problems , be it the machine or your machining approach