Tool holders are consumable just like any other kind of tooling. On a long enough time line, all tool holders will have the same problem. You need to replace them from time to time, just like pull studs.
If it concerns you, toss them in the scrap bin and get new ones. Loose bellvilles could be part of it. A 30 taper has less bearing surface than a larger taper, so they will wear faster. If the fretting gets too bad, I suppose it could even damage the taper in your spindle.
-Wes