Quill could be a parameter (have you checked them against the list that came with the machine). Look at backlash parameters too. Could be drive tuning. Is the belt too loose?
How many places are displayed to the right of the decimal point? If 3 then it is displaying metric. If 4 it is displaying inch. Just because you are doing G code does not automatically put you in inch mode.