This is likely due to multiple infant needs being depleted. Sims have a 'priority' system:
*They will override idleness with autonomous actions (standing around a few seconds will trigger them to fulfill their current lowest need, and default to fun and/or social if all are full). *They will override autonomous actions with player forced actions (They will drop any autonomous action the moment you tell them to do anything else).*They will over ride player forced actions with game forced actions (They won't keep doing homework if the house is on fire, or if there is a death, etc...).
Infants are special in that any action they initiate will override any other actions, except game forced actions. Basically it treats any command from them as a player forced action. If the infant's needs are at half or less, they will autonomously 'call for help' from their caregivers. When you choose 'call for help' it will cause the sim you used it on to drop everything and address the infant's lowest need.
When the sim doesn't respond fast enough, the infant will call for help again from another caregiver, or if none is present, the same caregiver again. That is what is causing the Sims to pick up and drop the baby. The baby keeps cancelling out and reassigning tasks.
To end it, switch to the infant and use them to have the caregivers complete tasks. And do them one by one. For example: To be put to bed, use infant to tell sims to place in crib> Once fully in crib ,click infant and choose "try go night night." To get a diaper change, use infant to choose 'help' on caregiver, and usually, if there is no other need lower, they will change baby.
You can use the baby to get ANY sim they have met to care for them. Even if that sim doesn't live there, is just passing by and is halfway across the map.