I assume that the encoder has differential signals. That means that the signal is represented by the DIFFERENCE between the two values.
IIRC, the 465B is a dual channel scope. So, you should connect the A to one channel and the A- to the other channel. Then set the scope in A-B mode (I think that is what it is called). That will display the difference between the two channels. The ground from at least one of the probes should be connected to ground.
Make sure that the trigger is set properly (to trigger on the signal). Also, as the speed increases, you will have to set the sweep frequency higher. (I know this is pretty basic, but I don't know what your background is.)
Since I don't know the mechanical arrangement you have, I can't suggest a way to get to the A- signal.