very strange. Encoder initialize failed. Do You get this message when powering on the OSP?
It looks You have hardware encoder overtravel from what You told. The encoder has limits itself. If You fix the problem by moving the axis manuallly it looks like that's the case.
So You need to turn encoder ... one full turn must be enough, I guess and put it back.
Sure, You need to adjust system parameter axis limts after that. This will be the parameter setting job.