@Katiardguez
One other thing, if your sim has earned the Beloved trait, relationship bars stop going down so they never drop out of your relationship panel until they die or you do something to cause the relationship to zero out. I'm not sure if that applies to most of the NPCs because if you never introduce yourself to them, you shouldn't have a relationship above zero. One other thing that I've noticed is that if you are in a conversation with another sim that you do know and an unknown sim joins the conversation, a lot of times, you will then know them even if you've never spoken to them and your relationship scores will go up slightly and their profile becomes available.
What you describe about your sims knowing all the sims on a lot when you load into it, I haven't noticed that in my game. Mine seem to accrue over time as far as I can tell. I will watch for it to see if that is happening now though.
Since this is happening for you in both a modded and vanilla game, the only thing I can really think of that might help is to repair the game and reset your user folder. Do the repair first and then reset the user folder. Resetting the user folder is pretty easy. Just rename your current The Sims 4 folder to something else or move it to another location outside of the Electronics Arts folder. Start the game and a new one will be created. Once you've made and placed the sim, save and exit. Then move or copy your saves, tray, screenshots and mods folders from your old to your new folder. You might want to hold off copying the mods folder in order to test first just to make sure it's not a mod causing the problem. You just play awhile to see what's what and quit without saving when you're done. Then move the mods folder in and test again. Not much to it at all. Just takes a bit of time.
Hope this helps.