Try cutting the pocket conventional, and it'll probably be closer to the correct size. Climb cutting will push the cutter away from the MDF.
As for the lines at different depths, it's possibly due to flex, or, the spindle may not be perpendicular. The imperfections lead me to think its flex in the machine. Try ramping in instead of plunging and the marks will be minimized.
The rounded corners are probably from CV mode. If you try exact stop, they should be square. Faster acceleration will also minimize the rounding, or get rid of it. You can also set a CV angle limit, so that 90° corners are square, but greater angles will not be. This may shed some light on CV settings.