NPC Sim invades home - inconclusive Cause (Patch 1.110)

I went back to playing in an apartment recently and added neighbours for them to have little interactions, however, the second I placed them next to my family, they kept going into my apartment without even being invited or talked to. Thus I can't put the time in x3 when my household's asleep as my game consider the neighbours as being in my lot. And the worse is that these neighbours keep taking food in my fridge etc... And even when I try to make them go to their own apartment by playing with their householde, their considered as being in my lot and when I come back with my family, they stay, even if I made them go back to theirs. That makes playing a burden more than something enjoyable