Sounds like your collet assy has seen better days.......But.......To help yourself out........when you get a new reamer......take it and put it in a lathe or reversing drill press and run it backwards and using a medium fine India stone.....gently stone the transition area from the lead-in chamfer to the reamer diameter........what you are doing is lightly "Dulling" the actual part of the reamer that does the cutting..........now the reamer will not have a tendency to cut where it wants , but to follow the hole and "Actually" Ream rather than "Cut"...........
Ken