Are you using SolidWorks approved drivers, or latest from nVidia ?
Running Vista or Win7 with 64-bit gave graphic issues in 2009 and needed to start SolidCAM in "Vista-basic"-mode, but is solved in 2010.
I'm running FX4500 with 64-bit Win7 in SW2010 and 2011 + SC2010 and it runs like a dream. I never go besides SW approved graphic drivers (see SolidWorks 3D CAD Design Software) as it have caused much problems earlier.
It sounds like a graphic memory limit issue and there is the quadro card better then Geforce. See that the computer just running SW+SC
There is SC2010 SP3 just released also... Maybe it's worth a try.