When trying to do that tight inlay work, the only way to get it "perfect" (that I know of) is to compensate for the kerf. You'll need to figure out your spot size to do this, and some work on the drawings.
I usually break my drawing into pieces, and then offset the individual pieces by 1/2 of the kerf (inside piece bigger, outside piece smaller). Depending on thickness and focal length, you can also try cutting the inner piece upside down to compensate for the beams's cone.
If you're randomly getting spots tighter than others, it could be machine accuracy, or beam quality... and may, unfortunately, require a little hand finishing.
I've never seen laser cut inlay look perfect...