Hi Kong,
I mill aluminum on my drill press and turn it on my wood lathe. It can be done. Don't expect super quality and tolerances though.
My drill press is a 1970's bench type import. CTT is the brand. I use a cheap cross slide vise and I admit that that is part of my problem with inaccuracy. The other part of the problem is in the axial endplay in the quill. I find that it works best when the quill is fully retracted and locked down. Depth of cut is changed by raising the table. You have to be careful not to move the position of the table when raising it. Also you have to be careful about how much of a cut you take or how fast you feed the part. Flex in the machine is an issue also.
When I first started the MT #2 tapered shaft would fall out if I took too much of a cut. Some gentle tapping and a rust fit
seem to keep it in now.
But it has served my purposes so until I can get the metal lathe and mill that I want it will be OK.
Chris