Well I think I can help maybe.
So if the slowest speed on your wood router is 12000 rpm
The max speed is 20000 rpm.
cutting speed X 4
----------------- = RPM
The numbers are from my text book there may be slight deviations.
Cutting Speed of a HSS end mill in alurminum is 200.
Cutting Speed of a Carbide end mill in aluminum with about cut depth of 0.015 is 800.
So using a hss the diameter of your cutter would have to be 5/64 HSS end mill running at your lowest speed setting.
If you got yourself a carbide end mill of 1/4" at the lowest speed setting the cutting speed formula works out.
So your problem most likely is spinning the cutter too fast. Your easist solution is to get a proper sized end mill for the rpms of your router.
You can also rearrange the forumula as
CSX4/rpm = diameter to figure out a good diamter.
watch your depth of cut as well.