Brass like to grab long cutters. Try a three flute endmill if you have some. The core is a bit heavier than a two flute, and should still give you alot of chip clearance. You may also want to look at solid carbide. Choke up on the cutter as much as you can this will help some of your vibration.
Another thing to consider is your fixturing. How are you holding the part. That has to be rigid as well.