Forgive me because it's been awhile since I have had to set one of these up, so I might forget a few things.
Have you adjusted anything yet? R6 should give you some sort of change in how it behaves. Usually "stiffens up". A lot of times, this is all you have to adjust. Don't worry about R5. That is just an indicator to show where the slowdown should start. In position and In Position Fine are just that, you can adjust where your spindle is positioned for your tool change (within a certain range). If In Position is too far out of position, then it can oscillate just like you describe. If adjusting R6 will not keep your spindle from oscillating in orientation, you may have a bad sensor. Make sure you take your orientation card out of test mode and try to give it an M19 in MDI. Sometimes I have had those things act weird in test mode versus normal mode. Good luck.