I hate when that happens...
Some of your oscillation problem my be in the tuning of the servo amps / system. Typical output for an SN7404 (VOH) is 3.4 volts. Same for the LS and S verstions. I had to think to remember what the SN stood for..
Might try a dual op-amp that will work on 5 volts single ended supply to get your inversion to go to the supply rail.