Well, you mention 1/4" hex, the first thing I was gonna say was guide bushing tightness but since you say its completely round, then I am guessing that this diameter you speak of is one of the smaller ones? How is the roundness, or I should say the concentricity of this diameter compared to the others?
Are you using righthand or lefthand tooling to cut said diameter? Sounds to me like the material is pushing away from the cutter. If feeds and speeds havnt helped then maybe you should look at the tooling youre using? If its not razor sharp that could be a reason why, then again I dont know the job or what the part looks like so Im really just guessing here. Forgive me if I seem like I am just going over what you probably already tried, this would just be my next route of investigation.