Random Townies keep walking in my house. My Sim doesn't even know most of them, but all of a sudden, they're upstairs on the computer, watching TV, and walking through my home. It has been happening ...
Ideas
You didn't mention which world this is in but I think this is common behavior in Sulani.
One possible solution, no matter where your Sim lives, is locking all the outside doors to everyone except Household Members and Service Sims. (You probably don't want to pay for a maid who can't get into the house.)
Another solution is to introduce yourself to the offending Sims, form a group, and then travel somewhere else. Disband the group and return home, leaving them behind.