You say you're taking a side cut - are cuttingthe same way in both directions (ie. climb cuts both ways or conventional both ways)? Climb and conventional milling will leave different side finishes.
How are you holding the delrin? Can you post pics of your setup & finish?
If you haven't already you may also want to read this.