Kdoney,
I would be hesitant to do this for several reasons, but understand that I am thinking of this in terms of my experience with metal working tolerances, which are often quite tight:
1) expense versus benefit: its going to cost a fair bit if you want to make some kind of quickchange that is accurately repeatable. It will take some work to make a quick change connection that does not introduce extra instability and off-center loading of your machine axis.
2) the inconvenience of programming for spindles that do not locate in exactly the same place. You might be able to work around this if you create some kind of a "homing box" with switches that are actuated by contact with a standard pin installed in that particular spindle chuck. Otherwise, you'll be edgefinding the part all the time, or you'll need a controller with capability to use work offsets (G54 to G59), or use a variable G92 work home. It gets complex to keep track of.
3) Ultimately, we are all lazy, at least I am Too lazy to switch heads and go through the setup procedure just to do one little thing. So it has to be really, extremely convenient to make the changeover, or I just wouldn't bother.
Have you considered making (or having made) a single high precision spindle, with optional provision for quick changing the cutters? I say high precision spindle because you need to use super precision bearings to control heat buildup.