Does it sound like the one in the attached link?
[nomedia="http://www.youtube.com/watch?v=u8WRfgngBrY&feature=related"]YouTube- CNC Industrial Hobbies Mill, 5/8" Indexable Endmill Surfacing 6061 Aluminum[/nomedia]
Not as bad as a cement mixer however it is loud.
Converting to a belt drive would quiet things down a bit, thats for sure.
You may want to make sure the spindle housing is clean and not contaminated with casting sand.