Not CAM related, but if I were a math guy looking for software to write, I think I would take a crack at the PCB autorouter market. There has been a lot of consolidation in that market, and there are only a couple really good packages (extreme $$). Take a look at www.connecteda.com for one of the new upstarts. Much of what is available is marginally useful.
Seems like there is already a saturation in the low end CAM stuff.