sounds like the motor is not getting the 3rd phase. Make sure you have a straight path for all three phases from the converter to the motor.
check the output of the converter. 3rd leg will be unstable but measurable
mebee see if you can borrow some 3 phase item to test it with