This happens in my current family's apartment as well. It's not just neighbors, though I have noticed that when you talk to neighbors in the hall they seem to take that as an invitation to come inside. I don't mind that so much. But what kills me is the random NPCs coming from the street below and just waltzing in like they know everyone. When this happens there are no notifications like "hey neighbor. I'm bored. Wanna hang out?" Like you get with your actual neighbors.
The Sims that come from down on the street act like they have been invited in or as though they are in a group with your Sim (I have noticed that if you do add a Sim to your group that does indeed invite them to your home automatically). They just wander in and sit down to watch tv or help themselves to food from the fridge. It's really annoying. You can ask them to leave, again as though they had been invited in the first place. I should also add that on my family at least it isn't a daily occurrence. It seems to be triggered randomly, but when it is triggered, however that's happening, it comes in a flood of 6-7 Sims, many times even the vendors from the food stalls are coming up.
At at first I thought it might be because a had a bubble blower close to the edge of the apartment and I thought maybe other Sims were trying to react to it like they do, but in my observations the Sims walking into the apartment are not reacting to the bubble blower, they literally just walk in and do whatever as though you had invited them over. I can't figure this out.