I don't have my black reference book of successful cutting parameters with me but, you're going the right direction. Chatter is too much load so the boring bar, toolholder, frame of the machine, etc. Whatever can't handle it. OR there is too little load, so the tool bounces along the surface and does not cut. But .150 DOC at 36 IPM sounds pretty good to me on a 1/2" EM and yes, slotting is VERY severe service for any cutter.
You might want to try corkscrewing (helixing) into the pocket if your programming system and controller parameters support it (or the old way - drilling a starter hole and dropping in on center of that!). On the helixing, if you make the Z advance light enough and keep the helix tight, the chips are light from the light DOC and there is always chip clearance out the open side of the cut, because if you cut a helix around a .375 base circle, then .125 of the cutter is always in the clear and its almost like side milling.
Oh one last thing, a set screw holder gives less support than a collet, but you have to weigh that against it spinning in the collet or pulling out with aggressive millling. That's why they came up with milling chucks, hydraulic holders and thermal holders - best of both worlds but expensive.