I have had very good luck in a precision sheetmetal shop environment using Weldon fluteless countersinks for deburring with both aluminum and steel parts.
You can mount the c'sink bit in a hand drill and make quick work for touch countersinking the burrs off. Most other bits with flutes will chatter and these Weldon bits cut a very clean edge on the burred holes. You can also make a small hand held arbor to hold them and do holes by hand in a twist when there are not so many holes to do. There are also some whip type deburr tools that are shaped something like a hook that do a fairly good job but for me they are somewhat awkward to manipulate.