Personally I use a pry bar between the spindle and the tool. Don't like hitting anything and with my luck I'd hit the tool and break it!
Mike
P.S.: I also pry the tool slightly without hitting the release button, then you just hold the tool and take it out! Even though the tool is being held, it is not that difficult to overcome the washers with a nice pry-bar.