Townies Keep Coming to Door

One after another townies keep coming to my door and the only option is only 'Invite In'. I can't even use shift click or MCCC to make them leave, I have to invite them in, which then allows other random townies to walk in without an invite. The only thing that has worked is fencing off my house and locking the fence to everyone but the household.