Soft woods tend to be much more fuzzy than harder woods.
What stepover are you using? Usually, with harder woods, and 8%-10% stepover, you won't have much fuzziness at all.
Cutting with the grain direction can also help.
I'd also increase your rpm to at least 15,000.
I use inexpensive ballnose bits from Drillman1 on Ebay.