I don't have any experience with belts but it seems that in one direction you are pushing against the sprung area and the other direction you are pulling against it. Think that has something to do with what is happening.
If the below crude drawing, the drive motor is mounted below and centered between the idler wheels, tension could be adj by moving the motor up or down. Or mount the motor on a pivot point and spring load it down keeping it close to the center if the two upper idler wheels.
Then again someone that is wise to belts and pulleys will respond pretty soon. Good Luck