I think it has two possibilities. There is a software problem or a hardware problem, or a 3rd option, both.
Try to do one thing at a time.
Jog 300mm in Y ony, then measure it. Now jog X 200mm and measure it. In other words can you jog the machine using the same dimensions around the rectangle and end up at the start position? This would check the mechanical behaviour of your machine.
Your diagram suggests that both X and Y axis are moving at the same time, or a small offset is added somewhere.