Damn tight is what I use. Most likely you cannot actually use a torque wrench on it unless you have a special socket to fit the nut. Maybe your collet nut will allow for that.
In such a case, it might also help to carefully degrease the inside of the collet and the tool shank. You can swab the inside of the collet with a Qtip moistened with alcohol or mineral spirits. I don't say that you want the outside of the collet to be that thoroughly degreased when assembling, it may damage (score) when tightened up. Just the natural grease off your hands is likely enough for the outside of the collet.