I was thinking that deflection would be greater with a spiral-flute tool than with a straight-flute one, but it shouldn't be that much. Grain direction might account for some of the difference you're noticing between cutting on the two different axes, or it might have something to do with your machine. To eliminate that possibility, try rotating the wood piece 90 degrees and running the program again, measuring to see if the error moves or stays with the same axis. You might also check to see if the spiral bit is really the size it claims to be.