Ideas
I have the same issue. Random townies comes to sift through my trash, then kick over the trash can which isn't supposed to be able to get kicked, and then they walk away. They walk into the lot to use swings, chessboards and any other activity related thing you might have (Today I had 3 random townies invade my kids tree-house), that is usuable by adult sims. They also sometimes, but not AS often, walk into the house and do activities like dancing or watching TV.
This is -only- in houses, and only way around it is to fence up your entire lot and lock it for the household and employees. Give access to a random townie you want to visit your household? They WILL come back and randomly invade your lot for activities later on. It is annoying, and it is annoting that you have to micromanage constantly, if you try to get some peace by fencing in your lot.