I assume you are getting oversize holes, Bill?
One thing: you need a very accurate collet chuck to hold a reamer straight (as if you didn't know
). A regular 3 jaw drill chuck is not likely accurate enough.
Thing two: the closest fit I have found with reaming is created when using WD40 for a lubricant. It will cut .0005" smaller than most any oil, tapping fluid or synthetic water based coolant.
Thing three: try using a floating tap holder to drive the reamer. If it will enter the hole properly, it should follow what was there. I'm referring to something like an ER16 collet system for tooling like this.