Are you conventional or climb milling? I get that sort of finish without cutting fluid on aluminum when conventional milling, but a climb milled surface usually looks a lot more shiny. It's shining both ways with fluid, either flood or just a few drops of Tap-matic.
BTW, what size and type (carbide?) of EM?
An aside - if you are edge milling, a 4-flute EM will allow you to use faster feeds.