Maxi - swarf inside the collet is all part of the job!
only cleaning out a collet once a fortnight is excellent - you should be stripping down the collet once a week to clean it or you are inviting trouble with wear and tear.
depending on your material you could do serious damage to the mating faces of the collet and spindle nose.
Some ways to reduse swarf could be to direct a coolant nozzle more into the collet that flushes out chips and try if you have room to adjust the tools coolant nozzle to spray the chips away from the collet.
both will only reduce your problem, i doubt antthing will solve it other than a regular clean down. which should take no longer than 10 mins.