It's probably not a problem with the endmill you're using, but your home-made machine's lack of rigidity. Try this - with all the motors locked up so you don't backdrive the axes, lean on the spindle in various directions, and note the deflection (a dial indicator can help with this). If you see more than a few thousandths (or hundredths, if you're in mm) of motion, then it's likely the tool is bouncing around as it tries to cut, which leads to poor surface quality. If you post pictures of your machine, perhaps someone will weigh in with suggestions on how to beef it up so this is minimized.