Whew, I'm glad you found a backup file!
I'm no XPert
but what I would look at is to examine the two solids you are trying to subtract from one another and see if there are any logical reasons why the operation would fail. From past experience with other older cad programs, whenever I got such a "Hu Jass" increase in file size, I knew I had done something illegal, and it was necessary to edit entities or surfaces so they
could trim correctly, or try the operation with a modified approach to my original method.