I assume you are using a hand held router?
I have had this problem in the past using a 1/2" end mill. Found out the collet chuck was worn out on my router... it was pretty old at that time. But, I found that by using a carbide tipped router bit I could get away with using the old collet chuck. I found that I couldn't take a very big cut (I was edging some 1/4" material and could only shave off about .010", full depth, at a time). I later learned from someone I respect that the helix on end mills makes it difficult to hold them in a standard collet chuck. Don't know if this is true but he seemed pretty sure.
Anyway, make sure to use the right lubricant, no matter what. It'll save your bits and ease your cutting. Once upon a time I was a fitter in a place that made aluminum horse trailers. We used a Craftsman 3 hp and trimmed all our sheeting to the extrusions they were welded to. Even used a 1/2" round-over! We always used A-9 or Alumicut. Night and day differance in cutting compared to using fluids normally used to cut steel. (WD-40 )
Hope this helps, be safe and good luck.