Graham, I pulled the documents on my xylotex.
The version was the latest 12 months ago, 4.03 onwards.
It is rated to a max of 30v but recommends 27v, I heard 425oz motors need much more so I maxed power at 29v with 4 CPU fans. (the case gets ice cold)
Now the Vref is a max of 2.0v, the earlier model was 3.6v.
If all off this is fine then all I can think of is step/dir are wired in reverse and you have tuned out the motors wrong. Emc2 and Mach3 are are opposite configurations.
It has something to do with square wave being a little bit out.
Just recheck your RS232 pin numbers.
Re read all documentation (xylotex manual, mach3 user manual).
It should be clockwork.
Another thing, 425oz motors seem like a glove fit for a G250 gecko, xylotex is a little to small. Without a voltage kick I heard 425's are lacklustre.