What's the goal from this, having repeatable Z height, just not having to swap collets when you switch to different sized bits or something else?
If it's just to not have to swap collets, you could keep using 1/2" tools directly in the collet, but use ER16 extensions for 3/8" and ER11 for 1/4" tools to keep the overall length a bit shorter.
Biggest downside I see is adding a couple inches to the length of the unsupported spindle. Defletction of a beam is proportional to length cubed.. By using smaller extensions, this is reduced a little, and of course 3/8 and smaller tools tend to exert less radial force on the spindle.