1st rule of chatter.... slow down the rpm, or increase the feed.
But that presumes you're cutting as close to the chuck/collet as possible, and the tool is clamped as close as possible into the holder. It's a rigidity thing. And, the tool is on center to .001/.002 above center.
My first take on this would be to do a little of both here... I think you're in the fine-tuning regime.