welcome back to the game. I think you'll find that things haven't changed as much as you might think. Mach3 is still a very viable piece of software at a good price point. Let me see if I can address most of your questions with a rant...
As of yesterday I just got my mill (PM-45) under motion again. I have been using the closed loop steppers from automation technologies for several years now with great success. They are fast, powerful and they just worked right out of the box without having to tune them. Very user friendly and reliable. I run them in conjunction with an ethernet smoothstepper and mach3. Everything plays nice together and works every time.
I recently decided to do some upgrades on the mill. One of them being a switch to linear encoders. It was just yesterday that I got one axis moving again using the linear encoders, so i can tell you that it does work...IF..you are able to connect to the drive and change the firmware. You'll have to change the encoder count in the firmware otherwise you'll get a positioning error anytime you try and move the motor. I am able to connect to the drives with the leadshine protuner software. I can't say that it will work for all closed loop stepper systems, but it connected to my drives with no issues.
So with all that said, Yes you can use linear encoders and I believe they are superior to the rotary encoders on the motor, but they are a lot more work to install as well. The feedback loop is done at the driver level. Mach3 is completely ignorant to the true position of the machine. All the drives that I have seen, mine included, have a fault output signal. So if I get a positioning fault, my driver sends a signal to mach3 telling it to stop motion. It's not as good as feedback control right at the controller, but it's certainly better than having a dead axis while the others continue to jump around.
If you wanted to do the loop control at the controller level you'll want to take a look at the klop board from dynamotion.. It has encoder inputs and I believe you can use standard stepper drives as the controller does the position monitoring. Dynamotion has their own software but they also have a plug-in for mach3 if that's the interface you are used too. If I were to do this all over, this is likely the approach I'd take but I have no reason to change as my current system works great. People also use the Galil boards, which I guess you can get used on ebay for a decent price, but I don't know a whole lot about them.
As far as sizing the motor, i think 1700 is a bit oversized for your machine. Mine are 1128 and I think they are likely a bit overkill.
Hopefully that answers some of your questions.