Like shred was saying, my machine is a BT30 taper and I'm using set screw style holders (I own a couple of the Maritime holders btw, so far I'm very happy with them!). I can't find a max tool diameter, but I've found a lot of people using 1/2" endmills happily.
I got some speeds and feeds from some other 770MX users that have the 1/2" version of the tool I'm using, so I'm going to see if I can compare at all. Bob W, the guy that makes GWizard, sent me a couple of emails suggesting that I contact Tormach (I sent him screenshots of my cuts that stalled) as he thinks the machine should be capable. But after reading more people's best practices with the 770MX and similar end mills, I want to try some more testing. I'm just miffed that Tormach gave me such a glowing recommendation for GWizard and sold it to me with my machine. So far it seems to just be the source of all of my trouble...