Can you try to identify the contributors to the flexing (i.e. router bearings, router holder, Z-plate, y-profiles, riser, bearings etc.) by measuring the movement of the individual parts when pushing the collet? That could give you a clue where to beef it up.
My bet would be the Y-profiles flexing/twisting and you could improve that by adding another set to the front of the risers and connecting them with the existing ones to a box with aluminum sheet.