If you or a friend have a lathe, try skimming just enough plastic off the surface of the idler pulley to true up the diameter. If you made it per the Solsylva plans there is probably a small valley between the bulges where the bearings are located. You can lay a straight edge across the idler and see that it is uneven. A sharp lathe bit will cut it nicely. I did this to mine. The belt will move one way or the other a little depending on the direction of rotation but has not given me the problem that you describe so far, and I don't have nearly the run time on my machine as you.
The bolt that the idler pulley is mounted on must be bent in which ever direction it takes to get the belt to run with minimal drifting.
If I ever have this problem on my machine I was thinking that I would make a new idler pulley that is shorter and has washers for flanges that would limit the belt wandering. (If all else fails, cheat.)
Check your gantry to see if it is square with the side rails on both sides in case one side is leading the other. This will cause binding.