The point you made on the build quality and stiffness of the machine is the one that has made choosing a spindle so confusing. From day one the advice I have received is to make the machine as strong, stiff, stable and robust as possible for matching aluminum. I have done my best to follow that advice for the frame, gantry and table etc. following that advice for the spindle is not as easy though.
when I looked into what "high speed machining" gear companies were using in larger mold shops, it generally wasn't the router type spindles that we are talking about here. Their 40,000 rpm spindles often look more like this:
Vertical Machining Center Hyundai-Wia ‎HI-MOLD560 - CNC Machine
It's often the guys that use kit like that which are the ones who will tell me that you can't achieve a good finish on aluminum using a router spindle. It's not because they aren't fast enough. They say the bearing and build quality is not robust enough to properly handle the vibration and other extreme forces involved in metal cutting and they include aluminum in that assessment. Again, I'm not saying they are right (or wrong).
I do however get a sense that cutting aluminum using a CNC router spindle is not their core purpose. It's often pitched as more of an afterthought like it can handle a bit of that if you don't push it, or "some light aluminum engraving" after "wood and plastics".
There are also the Datron type high speed spindles which look more like the router spindles we are talking about here but they go on $200,000 machines and they seem to be in a different league.
Anyway, all that is academic because I can't afford a 40hp 40,000 rpm machining center milling head, or a Datron 50,000 rpm milling spindle. In my price range, it's about finding the option with the least amount of compromise while accepting that there will be some extra hand finishing to compensate for my machines shortfalls.
As you mentioned, sacraficing some roughing speed seems like an acceptable compromise for me as I am not looking to start a mold shop.
I have some ideas for adding a second low speed spindle at a later date too, if it turns out that I need the low speed torque. Low (ish) cost BT30 and BT40 mechanical spindles and gear boxes seem to be freely available on eBay...