Hi,
a regen braking resistor is place directly across a the DCLink by a MOSFET or IGBT switch.
I assume your machine is three phase? Is it 380VAC? Lets assume for the moment that is is 380VAC. The input AC is rectied and smoothed by the DCLink
capacitors to 380 x square root(2)=537VDC.
If your braking resistor has a resistance of 100Ohm and its switched across the DCLink then the power dissipated in it would be 2883W or nearly 3kW. It will get hot quick, real quick!
In reality the braking resistor is switched onto the DCLink only for very brief periods, say a few seconds while the spindle is decelerating.
It sounds like your machine is switching the resistor on but not off again. I would guess the MOSFET/IGBT is faulty, or the circuit which controls it.
In lager machines its quite common for the braking MOSFET/IGBT to be on a separate board so that they might be replaced as a spare part, they have a reputation for failing.
You should replace the resistor now too. It will be unreliable after catching fire.
Craig