You didn't say what size end mill you are using, so the speed and feed thing is hard to say.
5000 series aluminum is fairly soft. It is not normally used for machined parts per say. 6000 series is harder, so it will normally finish better. and 6061 is more commonly used that 6063 for machined parts.
For fish mouthing a common hole saw will work quite well. Or you can get "rotabroach" mills, a high dollar hole saw. Both Hougen and Jancey make their own versions.