The Sims 4 freezes randomly only for one particular household

Product: The Sims 4
Platform:PC
Which language are you playing the game in? Polish
How often does the bug occur? Every time (100%)
What is your current game version number? 1.105.345.1020
What expansions, game packs, and stuff packs do you have installed? All of them
Steps: How can we find the bug ourselves? It happens very randomly. No particular interaction or other thing is necessary to trigger this.
What happens when the bug occurs? I have been experiencing this problem since I moved to another house with my main household. I have like 5 families across many of the maps, that I play at the same time. However only one seems to be cursed by some kind of a glitch. Whenever I load the household up, and play for a few minutes, game just goes crazy and keeps freezing and unfreezing randomly, unallowing me to do anything, as even when I click on something, no interactions pop up. After some time it goes back to normal, and then again, like a minute or so and it keeps coming back. My main concern is the size of my plot, as it is a very huge mansion with 5 sims living in it. However, I've played on similar plots and never experienced any of this. Whenever I change to my other households (like the children who moved out from the main), everything is perfect. No glitches occur. Only one household is touched out of like 4 or 5. 
What do you expect to see? No freezes and just a smooth game
Have you installed any customization with the game, e.g. Custom Content or Mods? Yes, I've got a lot of mods, however most of them are cosmetic. I have only like 5 mods that use scripts.
Did this issue appear after a specific patch or change you made to your system? I'm positive that it didn't have anything to do with it. I update my game whenever an update is ready, however I have been playing for some time since the last update and I haven't had any glitches before.