Sounds like the quadrature signals are not clean, I would check these things:
1. Is 7I33 jumpered properly to match encoder type? (TTL or differential)
( using TTL encoders when the 7I33 is jumpered for differential will
generate the type of behaviour you are seeing)
2. Measure 5V power to encoders
3. Check encoder wiring carefully
4. Trace the encoder signals from the encoder to the 5I20 either using a voltmeter or LED and resistor.
5. View the encoder signals with an oscilloscope if you have one.
Peter Wallace