There are 3 flute drills that are designed to make a square(ish) hole. I have never used one, but I think they need to be able to walk around a bit. You might look up some Youtube videos on this subject.
With an end mill, you can not make a corner radius that is smaller than the end mill radius. The only way to really create a square corner is to broach.