Hi Frank. Fascinating projects, these spindles.
Let me hazard a response on a couple of your questions.
First, why harden? I think the answer is to make sure the taper is durable. If you score the inside taper, or raise a burr on it during a tool change, your taper may be ruined, or at the least may need refinishing to be accurate. You have to consider the possibility of a chip getting stuck between a tool holder and the taper. These are conditions that can ruin a spindle if it isn't hardened at least a little bit.
With that said, it isn't clear to me a lot of the import mill spindles are hardened to a very great degree. YMMV, but if it was me and I had put all the work into a spindle, I would want it to be hardened.
Making a hardened spindle does complicate your life a bit. You have to do the heat treating (or work with pre-hardened stock) or send it out. And since the heat treating will cause the workpiece to shift a bit, you have to do the final machining in the hardened state.
Traditionally, this would be done by grinding. You could use a toolpost grinder on a suitable lathe for a project like this. You could also look at the possibility of hard turning with a PCD or CBN insert.
Incidentally, your retention knob grabber looks like it uses spring fingers rather than balls, so you need to figure out how to harden it anyway so it stays "springy".
Second, bearing preloads. In generally, you're dealing with a trade off between the rigidity of the spindle and how many rpm you can turn it. Higher preloads will limit the rpm of the spindle.
This starts to be a bit of a black art. There are no hard and fast rules here, and a lot is figured out by experiment. Your best source of information is the bearing manufacturers. They'll tell you all there is to know about how to fit the bearings and which bearings are best suited to which applications. They even have sections on spindle design. Be sure to go check all that out before you go too far.
I've been collecting links and notes for my own spindle project for quite some time. Perhaps some of it will be useful to you:
In particular, see the comments on Frank Mari hard turning a spindle in the VMC.