Dust collection is designed to capture the dust before it gets into the room. Once it's in the room, it's too late. Trying to vent the room will never do a really good job, as air flow will probably run from the room entrance, to the exhaust fan. Dust will still settle anywhere out if this direct path. And if you're only talking a 4-6" exhaust, I doubt it will get much at that. Imo, you'd be much better off trying to get the dust before it leaves the table saw. I have a small 1HP dust collector on my table saw, and you'd be amazed at how fast the bag will fill up. Without the dust collector, probably at least half of that dust would be spread throughout the room.
Having said that, it's very difficult to catch all the dust from a table saw, because maybe 20% will come out the top off the blade.
One other thing, dont ever cut a 4" hole in a support beam.