You are asking how to make a square hole. This is a difficult task as when you machine something on a mill you are using round endmills.
Now it sounds like you want to make something similar to a mortise and tendon joint to attach two pieces of metal.
You might want to think about using a small endmill say 1/8" to cut both pieces. If you design it correctly you can use a small radius on both the male and female sides and they will fit together without any filing. You need to keep in mind depending on the accuracy of your mill you will need to allow for a friction fit. This basically means that if you cut say a 1" square post and tried to put it in a 1" square hole they would be very tight and hard to put together, so you allow for this my undersizing one side by .003 to .005". Then they will slide together the way you intended.
I don't know what your trying to build but you might also think about using a couple of tooling pins and a counter sunk socket head bolt to tie the pieces together. The tooling pins are a tight tolerance fit that ensure the pieces will align perfectly and the bolt provides the strength to secure the connection.
Hope this helps.