My go-to idjit start point on drilling a quarter inch hole in mild steel is 1000 rpm and 3 ipm, with a 0.1-.3 or so peck and a G83 cycle (full retract) if I'm running a program, with cutting oil. For a few holes, coordinate drill with the shuttle. Double rpm if the diameter halves, etc. This totally ignores sfm, feed, and all the other stuff, and all the tables. Drill in a chuck or ER20. I used to get all knotted up in calculating the theoretically correct numbers, which sometimes worked and sometimes needed tuning, and decided that I needed a standard answer that didn't strain the machine, burn bits, or require an hour of messing around for a 2 minute hole. Mostly, this keeps me around 60 sfm with a thou chip, and my bits last 25-50 holes or so between grinds.
Yes, I know this is offensive to the gods of CNC. And usually wrong, formally, and would get a really, really dirty look from anyone doing anything like production. I'm not.
All that said, the next (actually, first) thing to check is whether you have a good quality drill (PTD, Cleveland, etc- not box store) and that it's properly sharpened. I generally use a 4 facet style grind now; the drill really looks like a 2 flute mill. Drill-doctor also works. Do not assume that a new drill is sharp, accurately ground, point centered, or has a relief. Check. Or just regrind the minute you pick it up.