What's too fast ?
In my daily travels, I see lot's of folks that could produce more parts in a given amount of time if they would just utilize radial chip thinning, smaller roughing end mills (increased rpm = more IPM), variable pitch end mills (reduced chatter), etc.,etc.
So I say if you have the HP to pull the cubes (cubic inches matl.), go for it.
Friend of mine always says, the only thing better than cubic inches, is cubic feet.
Steve