How does the tool look when you stop it, is it plugged up (gummed up) with aluminum? Ball mills cannot successfully plunge very far, you would have to ramp into the cut and should never go deeper than the radius of the ball. Even that is pushing things if the cut is full width of the ball, most likely the tool will plug solid.
Try a bit of WD40 on the surface next time. This will usually prevent the tool from plugging up, but a Trico microdrop lube unit works wonders for milling aluminum and clearing the chips from the cut zone on a continuous basis. Trouble is, the Trico thingy might cost as much as the X2 mill