How to keep neighbors out of your house?

Mine keep coming over even when I'm not there. Is this a bug? They shouldn't be allowed to do that. Maybe they're smarter like they said, but it's silly when they come into my house when I don't want ...