My first suggestion would be to re-verify all your new and modified wiring. Double check that every wire goes to/comes from where you think it does, and also check voltages/states where possible.
While you're doing that, look at what generates noise and where. VFD drives are electrically very noisy, and can generate spikes and phantom signals in nearby unshielded wiring that can cause encoder feedback and drive command signals to go wacky, especially on unshielded single ended connections.
If you look at your VFD's install manual, it probably recommends a line reactor on the input side and possibly a load reactor on output, plus an RFI/EMI filter somewhere in the middle.
Lots of people put the VFD in a dedicated metal enclosure as a faraday shield, and put a fan in there for cooling.