The gearing may not be an issue. I didn't even concider that you would have a 36V power supply on a machine that size. You need around 75 volts to get the performance out of the motors, and of course compatible drives.
I would install these drives, these are the ones I am using on my router
https://www.automationdirect.com/adc.../STP-DRV-80100
The drives are fully programmable, no DIP switches. They can be programmed to work with your existing motors. The goal is to keep the stepper motors at <800 RPM at max speeds, so changing gearing might be needed.
and this power supply
https://www.automationtechnologiesin...vac-or-230vac/
The other option is to go with ClearPath SKSD servos, and the above power supply. I retrofitted a customer machine with that system and had very good success.
https://www.teknic.com/products/clea...sShowPopUp_100
The motors have built in drives, so no external drives needed. Just connect the motors to the power supply and your breakout board.
I think Mach3 has the ability to un-slave the motors for homing if they are wired to separate outputs. Not sure about that. I hope that one of the Mach3 gurus will comment here.
Before changing out the control system, it might be best to get the rest of the system up to spec first. Then if you are still not getting the performance you want then is the time to look at a different controller. Mach3 is a popular system and there is a lot of support for it, and more importantly you are already used to using it. You might be pleasantly surprised once the mechanics and electronics are working as they should. There are some better options for Mach3 breakouts than the parallel port, but you don't want to make too many changes all at once, one step at a time.