Hi Jim,
1) Most ball nuts use 2 different size balls, alternating big, small, big small etc. This is to allow the smaller ball to roll between the bigger balls, without rubbing the screw, to reduce friction. It will be hard to tell the difference in size. I think its just a few thousands. Usually you need to buy a new nut if the balls come out. You could try measuring a few to see if thats the case for you ball nut. How does it work on your screw since you reasembled it?
2) As for the table rails, on my last machine I mounted everything as close as I could but made sure the bolts I used allowed me to loosen, tweak, and retighten. I also mounted the bearings to the table as parallel as I could, inserted the rails etc. move the table to the extreme end of travel, bolt down that side, then move the table to the other extreme, and bolt the other side.
3) Make them from wood for now. What's the rest of the machine going to be made from?
4) the software can compensate for tons of backlash, but it's better to build in as little as posible. I had .006" on the y on my last machine and it was annoying because I knew it was there but it didn't affect the parts after I dialed in the backlash comp.
Eric