There's a great burr tool made by Shaviv, a ceramic blade, that I use a lot for teflon, and it doesn't dig in or gouge when used carefully.
I didn't think it would work, but it has turned out to be incredible for really soft stuff. It does teflon and delrin equally well.
For complex shapes, have you thought of dry ice and a tumbler? Years ago we tumbled punched dense foam rubber parts with an abrasive and chunks of dry ice. The flash from the punching operation came right off. (they were slot car tires....!)
....a place to start....
http://www.shaviv.com/shaviv-pdf/SHA...05-English.pdf