My suggestion is to first verify that you have Mach3 calibrated correctly by doing some simple tests.
In Mach3, zero your DROS, then go to the MDI window and enter some simple gcode commands like:
G21 (this will put you in MM mode)
Then measure the distance travelled with a vernier to confirm that it is indeed 100mm.
(you might want to get ready with the e-stop button if your machine tries to head for the hills ;D )
Then try your other axis (G1 Y100, G1 Y100).
If this doesn't do what you'd expected then you need to change your motor tuning parameters. There is also an automatic calibration routine in the Mach3 setup screen you might want to check out.
If the above all checks out, you could then create a hand coded gcode file using a text editor. Something like:
G0 X0 Y0
Try cutting that in some foam or MDF and measure the resulting square.
Don't forget to take into account the cutter radius. So if you used a 6mm cutter, the above square should be 94 x 94 mm.
If that all checks out then it's time to start checking things in CamBam.
I hope that helps and good luck!