My number one annoyance in the game is text messaging and phone calls. I've set my phones to 'silent', which at least stops the game from reverting to slow speed every time a call or message comes up, but they're still extremely irritating. And I end up missing the one call I want my Sim parents to get, the calls from the school. Even death doesn't save me from them, since the ghosts continue to message my Sims. This is the whole reason I won't buy GT, and one of the reasons I wouldn't take GF if you gave it to me for free. I hate, hate, hate the calls and messaging.
Second would be the repetitive whims. I don't want to turn them off, because I do find them useful, but certain whims turn up way too often (the bee box!) and you shouldn't keep getting the same five whims when you cancel each one out.
