Unfortunately this is something that they were going to hit at some point. Though are sims and all the lots aren't loaded up or on the current lot they are still in the background being tracked. At the moment we have 145 lots we can move sims into and if you wanted to you could have up to 1160 sims in your saved households (housed sims only) with the expectations that more lots will be added so the potential of more sims being in the world also raises.
Personally I don't think it's the game itself since from the beginning they have said they are making ts4 as playable as possible for lower end systems so I think it's the lower end system that are struggling. I know that on my old PC after I got CL my game started to take a hit and I had noticable lag in it's hoods. I have since upgraded and though I don't have all my settings at max I am able to play the game much better then before. I think with how the raised their minimum specs requirements for C&D they are going to have to sacrifice playability on lower end systems continue to raise the minimum specs as they see fit. It's something that most gaming companies end up having to tell their consumers. Upgrade your system if you want to play are games.