I use Tormach 5" machinist vise. Perfect fit for Tormach mill, but your kind of isolated as most jaws and vise accessories are for 6" models. Some work on 5's but of course stick out.
I like how Tormach unit mounts with keys and no need to tram in line. larger vises will work fine they just have the issues you mentioned.
As for weight, I would not worry about that at all! My Tormach 4th axis weighs 185 lbs and mill performs better when its in place.
The added weight dampens the machine a little more. Not much but when you know the machine well you notice the difference.
Main thing with a good vise is for it to hold things flat! No jaw lift! My Tormach unit requires regular cleaning and careful adjustment to keep it accurate.
Weight of 5" vise is about 75lbs and at limit for me to lift. I use a 400 lb max ceiling mount HF winch on a trolley to move vise and 4th axis on and off mill table. "no full enclosure" If I had full enclosure a "C" lift arm would be used.
The trolley winches runs entire length of shop on ceiling in both bays and can be used to move and lift anything less then 400 lbs. Very nice to have imho when you have limited strength and a real back saver.