Hey Bear, well a modified printer is certainly an option. Since you need rotary engraving the rollers could be made to turn the tube while the head scans the length.
Yes, Mach3 should work ok. You probably need the laser (raster) plug-in which is available on their site. The problem would be the laser, your application won't need much power - so a small (25-40W) CO2 would work great or you could try with a high wattage diode laser (the driver PSU is always the problem here).
Cost wise, I would look at the laser and other components and see if it makes sense to build your own or buy a Chinese model or even a used brand name.
If the diameter varies, you may also need a Z-axis to control focus. I am sure Mach3 could handle this but haven't tried.