Sorry, I can't see your attachments (probably because you don't have enough posts yet). Are you trying to make sharp 90-degree inner corners? That would not be possible with a round rotating tool.
You can either use a smaller end mill to make the corner radius as small as possible, or overcut the corners like this: