How to stop random sims talking to you and invade your private property?

1) every single non-household sim considers their duty to invade into my lot and start walking around in my house. i locked the door for everyone but household members, but now nanny can't pass, so i...