"The next step may sometimes be to drill the hole undersize (by 1/64"), and then attempt to straighten it out by redrilling with an endmill. For this, I would recommend that you only drill a shallow pilot hole, because endmills don't reach very deep...maybe 3X diameter. So the endmill should help straighten out the first part of the hole, and if you then switch back to a drill bit that fits the endmilled hole, then you have created a sort of "guide bushing" to drill the remainder of the hole depth with."
1/64 under might be pushin your luck but that method works very well. Also there is nothing more important than a properly sharpened drill bit and using a centredrill to start things off.
By the sounds of your vise setup it is not to high tech. I really like these >
http://www.kanttwist.com/c_quick_holddown.html < for quick setup on my drill presses.