What kind of a machine are you trying to do this on? Something you built yourself, or a commercial unit? What kind of spindle are you running the tools in? What feeds and speeds did you try? How were you holding onto the workpiece, and did you notice any movement in it as the cut progressed?
ABS is a rather low-melting thermoplastic, so too much spindle speed will cause the material to soften and stick to the cutters, especially if you're trying a deep cut with slow feeds. Single or 2-flute cutters will work better than 4-flutes, and light fast cuts work better than slow heavy ones. You generally want to use the largest tool you can to get most of the material out of the way, before getting in there with small cutters to remove the last of it as a finish cut. Air cooling helps a lot when cutting thermoplastics, preferably with an air chiller powered by a compressor.