I have had this same issue as well recently, it's as if the entire lot is selected to come with my sim, where they have fame, no fame, whether it's an Infant or Adult. Anytime I travel with a sim that has household members, Including pets, are all following me to any lot, community or home residence. Babies and pets are stuck in hold or a crib will appear somewhere on the lot if it's an infant.
(Ex. Sim 1 went to work at the film studio and she has a pet and an infant at home. When arriving to work the pet was equipped to hold. During work, she kept trying to care for an infant I couldn't find, and as autonomy was on she was constantly looking to care for the infant. I then found the infant in between a fenced off area as if it were randomly placed anywhere on the lot.)
It is especially difficult to manage for sims with households of 2 or more, household member follow the selected sims to High School, Work, Dates, Community Events, and Private Events.
Sending them home should not be the solution, this bug is definitely overwhelming so I do hope it can be resolved!