What RPM is the Sherline capable of?
Any aluminum build up on the cutting edge?
Chips need flushed away more then the cutter may need lubed on very small bits.
Always use the largest end mill possible.
.050 may be a bit on the heavy side for a .050 and .5 IPM is on the low side.
You are not rapiding (is Rapiding a word ) into the part are you and then starting the .5 IPM feed?