It happened the third time now that my sims were blamed from their neighbors for being too loud at a late hour. The thing is, my sims don't do ANYTHING loud. Just now, my sim was just watching a video...
Mmm, I started another save where my sims currently aren't living in apartments, but I don't think moving out would help, I had two families where I had this problem after all, but I'll see if the problem is still there in the new save.