I would say your analysis is correct, the spindle is not aligned with the axis.
Here is how I measured mine. I took a welding rod and bent it as shown in the picture, mounted it in the collet. Then used paper as a feeler gauge under the tip. I shimmed the motor mount and adjusted the mounting as needed to get it in alignment with the table. The circle you can swing is large enough that any minor error out at the tip of the welding rod would not be noticeable in use.