How can I prevent my sims from leaving a lot?

I noticed that every time invite one sim to a lot, for example, the museum, and invite another sim to the same lot one of the sims will leave. This is annoying when trying to get sims to meet. It’s also annoying when sims are hanging out together and one just randomly leaves. How can I stop this?