Congrats on the machine. Over time this will become intuitive, but here’s the process…
1) this is all theoretical, its a rare home machine that can remove metal at theoretical limits, a practical stop either on rigidity or power kicks in first
2) determine cutting speed - function of cutter and work material, expressed in feet per minute - hss cutter into AL is about 400 fpm - this number is a plugged number, looked up from a reference like machinery's handbook.
3) from that determine spindle speed. its the max surface speed for the periphery of the cutter in that material/cutter combo
4) based on how big a chip per tooth you want to cut (maybe a thou or two for a small cutter like that, figure out the advance rate in inches per min.
I’m too lazy to do all the math, but those are the steps. you will likely not get to a theoretical maximum before the machine gives up - think of it as the machines limitation in horsepower and rigidity to remove only so many cubic inches of material per minute.. As something is going to be set less than max, I like to see a reasonable chip size and the spindle slowed down as this increases cutter life (its not a linear relationship).
if you are getting chatter, backoff on feed or speed or as you've noted use a more sturdy set up. overall, the numbers you posted sound reasonable - it worked didnt it?