Since you are from the US, and everything else you mentioned was imperial, that .05 you mentioned must be .05", correct. Fifty-thou step down with a 1/8 ballnose is way too much. Okay for roughing in MDF, but for finishing you should reduce that value significantly.
There may be more issues here, but that one jumps out at me as the first thing to correct.
Dan