.005 might be a little light. I would try to leave a little more say .02 to .03 for the final at least. You also don't say if you are climb milling or not. This will have a big difference on the finish. How new is the finishing end mill? Is it a dedicated finishing mill and not the one you used for roughing? Is it a long end mill? This will tend to vibrate or skip if long and light light cuts. Is this finish with all end mills or just this one? What type of aluminum? Why the odd size of end mill? Is this a resharp?
There are a lot of factors which can play into this.