Residence Keys?

Can anyone with a residence key to your sims house freely enter and leave if the door is locked? i cant keep it unlocked as every Tom, Dick and Sim just enters and does whatever they want, but then so...