The problem is, not everyone wants the same things. The Sims is so huge and diverse now that there are many, many themes and specifics that could be added to make different subgroups happy. I wasn't interested in fame at all, but there were people who really wanted it, so we got it. I personally didn't care for laundry or parenting or food groups, but those things were requested by other players and they got them. I did, however, make many requests for Vampires and we got those too, and some didn't want them at all. So pleasing everyone all the time isn't possible. But pleasing most people sometimes is probable. So I think that's what they're trying to do. Unfortunately, this also means disappointing some players sometimes, so we have to live with that.
Something like improved story progression (because it does exist in Sims 4, it's just less invasive and more subtle than Sims 3's version) is tricky because if they decided to do it, they would have to make sure that it wouldn't interfere with rotational players' games, so it would have to affect only townies, or offer a way for players to select which households to "opt in" so that their played households remain untouched. I, for one, definitely do not want the game to start making more changes to my Sims that I didn't want. I already hate when they show up in random job positions.
I do agree overall that there are still many basic features that we desperately need, like a Fear emotion, more NPCs, and a traits vs emotions overhaul or deeper personality system. Because without these things, no matter what content is added, the game will continue to feel incomplete and shallow.