I'm sure other's will have more knowledgable answers coming but the potential problems that springs to my mind right now based on what I've done is:
1 - you have very narrow cutters machining a very large piece of aluminium.
2 - the feedrate is very substantial, especially given the size of the cutter
3 - the pass depth and stepovers I would halve, especially again as you are using a small cutter
4 - the spindle speed may be too fast, heating up the cutter
5 - are you clearing ALL the waste as it occurs?
6 - I generally get better results from 2 flute upcut spiral cutters as the endmills tend to clog straight away, at least for me anyway!
From my little amount of observations machining aluminium, a smaller cutter being forced through large material at substantial depth passes would be flexing, and heating up very, very dramatically. With 1/8" endmills in a 4.5kW spindle, I imagine it as akin to sticking a toothpick in the barrel of a jack hammer and using it to punch holes in mud.
Seeing as you have a 4.5kW spindle, buy some 2 or preferably 3 flute upcut spiral cutters suitable for aluminium in a much larger diameter, so they won't flex, will cut more material, faster and easier. At least for the roughing anyway. Maybe split your roughing passes into one at 0.5 clearance for the larger bit, then 0.35 clearance for the smaller roughing bit, then take shallow passes at a slower feed for the finishing cutter but not so slow it creates heat buildup.
These are my observations, and they work for me, but may not suit everyone.