It must be an optical illusion, but that last picture of the part looks like furrows in a field
A 1/32 endmill is smaller than I have worked with to date, but I can imagine that it is extremely flexible. When flexed, the bottom end plane of the tool flutes will be tilted, and this will create definite roughness. I would recommend milling the pocket with a larger tool (perhaps 1/8"), then clean up the rest areas with the smaller tool. Of course, this might be a lot of cam work, depending on what you have available for software.