Sit down and work out what the surface speeds are in the areas that the cut is working for you, and calculate the surface speeds in the area that it is not.
As you approach center, the speed that the material moves past the cutter, slows, a lot. This slower surface speed causes the cut dynamic to change, the chip formation and surface finish change as well.
With a cnc lathe, the answer is to program for a constant surface speed cut, using a variable speed motor on the lathe. On a bench lathe, the use of a variable speed motor, and raising the rpm as you approach center, is an option.
Otherwise, it becomes a matter of finding that happy medium and making it work out.
Using a quality insert is a real plus too, if you are using inexpensive Chinese inserts.
Using a cutting fluid? Even applying some with a brush will make a difference, flood is better.
A sharp HSS tool can do wonders at slower rpm too. Hog off what you want moved, then finish pass for pretty with a HSS bit.