Is the part securely held down?
The machining characteristic of brass that one needs to keep in mind, is the tendency to work harden, release, work harden, release, over and over again.
Initially, the workhardened skin is created by the tool itself. So the material resists cutting, and then the next edge comes around and since the feed has advanced the position of the material, the cutting pressure is now much higher than it was, and the material skin gives way and the edge penetrates the work, but the chipload is much higher than planned for. The positive cutting rake of the endmill provides a wedge effect that lifts the work onto the rake face of the tool (inside the flute). This is called 'hogging in'.
This creates poor cutting results, especially on thin sections.
The surefire method to eliminate the hogging in problem is to use a straight flute tool, with absolutely no honed edge (ie, sharp, not brushed or overcoated). Typically, you can grind a little bit of the endmill, just inside the flutes, creating a flat facet that changes the geometry of the flute right at the super acute cutting tips. All you need is a tiny flat zone, that is radially oriented and parallel with the tool axis.
This same trick works wonders for drills when drilling brass.
You will also want to use a stub length flute on the tool, when working with a tiny diameter like that, to keep the tool stiff enough to consistently take a chip with every revolution.