How to keep neighbors out of your house?

I am thinking the reason that sims will come into your home 'uninvited' after you talk to them outside, is that maybe the game reads 'public space' as the property of the lot which is open. I remember a dev explaining how when your sims are inside they are aware of what is in the public space surrounding their home and will autonomously decide to go use that stuff. Basically, the neighborhood is an extension of your house, of sorts. I may not be entirely correct, but I think it is something along those lines.