I'm learning not to be too aggressive with my 770.
I ran "big" CNC mills for years before getting my own 770, so my original programs were based on high speeds & feeds (As I was used to doing).
Boy was I wrong!
You have to slow things down, and kind of 'baby' the 770.
There simply isn't the same amount of rigidity and torque.
Be patient, you'll find that sweet spot often by reducing the feed and rpm a bit.
Also, try taking light cuts and more passes in 1018.
You'll get it done. TORMACH's can make any part a 'big' mill can, it just takes a little longer with lighter cuts and feeds.
For your 3/8 endmill, I'd try 2000 RPM, .030 DOC, and .003 IPR.