Have you tried using an endmill instead of a router bit?
You can get them in 1/4 inch shanks.
Sherline sells some for one.
You probably won't have the rigidty to go deeper but you should be able to feed faster.
An X3 will get you much higher depths of cut.
Hoss