Can you use a solid carbide endmill? Your surface speed at 3200 rpm is 40 Meters per minute. You could go 100-150 M/minute with coated solid carbide. How many flutes do you have? 80mm per minute is crawling even with 2 flutes. A chipload of .03mm is safe with such a light depth of cut. That would give you 192mm/minute with a 2 flute cutter at 3200 rpm. Hang as little of the tool out of the holder as necessary.