The reasons might be because this type of SP would hinder performance. The game would have to keep up with too much to connect all those Sims and their lives and families etc. This is one reason TS3 has some performance issues is because of Story Progression. One way to solve it was to have the game move out families and give the player a notice. They were culled. But at least TS3 gave a notice but to be fair it didn't matter if that Sim or family was a friend of my Sim, so, there is the problem of culling in TS3, too, even if they were good friends of my Sims. :/
There was also a problem with TS3's SP where it would move in four or five other Sims (for no apparent reason) with one of my Sims living in a one bedroom home, if I wasn't currently playing that Sim (playing rotationally) and that was nonsensical to move in that many Sims into a one bedroom house. It would also involve my Sims in relationships that I didn't even meet their partner...if I jumped to play a different Sim for a few hours. Story Progression was not a good thing in TS3, what was good was the modder's mod to make it work the way players thought it should work. The TS3 story progression had way too many issues.
But the reason they might not have added it to TS4 is because Sims can just be deleted without any harm to existing Sims, that isn't true in TS2 and TS3, so all that telemetry data about those Sims would have to be considered, and again performance issues would happen if this game has to keep up with all that data.
Just another antidote, in TS3's story progression (unmodded) adult (child) Sims living next door did not get jobs on their own. This was only done if the player's Sim was in a career and the game called for a new townie to be a co worker...so there is also the problem that no, it didn't work like people think because in a vanilla game it was awful.