I machine a lot of Delrin, and I use a "Knife" style tool of high speed steel. This style tool has a long cutting edge and sharp point. It works great for Delrin, in my experience; leaves a smooth finish, and cuts like butter. Older lathe books show pictures of this style tool.
I have also used small insert tooling with good results, but the Knife tool seems best for my items. You may have to build some sort of back support to keep the Delrin from springing away in order to keep your tolerances, or perhaps a roughing and then a finish cut.
Good luck with your project! Joe