You are basically rubbing the material off. You report a better surface finish with a smaller diameter cutter so that hints at your cutter speed being too high. Can you change the rpm of your spindle? I would try about 10k for the 1/8 bit and 5k for the 1/4". Or you can deal with your workholding issues and increase your feed rate.
Do a bit of research into surface feet per minute (SFM ). That is how fast the cutter is moving wrt the material. Also look into chip loads.
The recommended SFM range for acrylic is 200-600 SFM. Your 1.4" cutter is running about 1300 while your 1/8" is about 650. The speeds I suggested put you in the middle of the range.
A general recommendation: Choke up on the cutter as much as you can. That will minimize cutter deflection and reduce chatter. Since deflection increases as the cube of the change in stick out, a little goes a long way.
bob