I had to stop playing my 100 baby challenge because of this. The baby daddies would come in uninvited and never leave. Can you imagine having 20 (or more) baby daddies hanging around? I was annoyed with two so I put a stop to that and will come back once it's fixed. For a while I found that adding them to my group brought up the option to ask to leave but that only works sometimes. Then I was using MC Command but got tired of fighting with it every day. Even if they don't like you they still come around.