Witness sims keep showing up

I realize there is a thread for this but it’s years old and I couldn’t find an answer. 

I just placed a sim from the library. The house already existed in the game. I just evicted the old sims and moved a new one in. But now all these witness sims keep showing up and destroying my house. I’ve had to fully security proof it with a fence and a locked gate as well as locking all the doors in the house just to keep them out. Even went as far as to add the lot trait for private dwelling. Nothing is working. After a period of time and conversation sometimes I can ask a sim to leave but they just get replaced with a new witness. First there were two and then I fought them and they left but then two more showed up and then I managed to ask them to leave but three more sims showed up. 

please help I spent so much time remodeling. Worst case scenario I’ll just leave it and never play this household again.