Plunge cutting is tough on toolholders because it forces the toolholder deeper into the spindle bore and the toolholder starts to self-lock.
It was explained to me that toolholders are designed to be held in place with drawbar tension . The taper is not meant to be self-locking; it is only meant for locating the toolholder accurately in the spindle bore.
I usually spray all my toolholders with dry lube before they go into the spindle. The lube prevents the self-locking.