I have this problem specifically when playing on a residential (or apartment) lot that is next to an area that spawns festivals. When the festival ends, the festival vendors head for the nearest place they can fill their needs (usually sleep or bladder) and that happens to be my house/apartment. I have to keep the doors locked when playing at those locations. Otherwise, they just walk right in (without knocking) and take a nap on my couch or start using the toilet. But when I play at a residential lot that does not have a festival area nearby, I never have this problem.