Do something else if you're finding that Infants are unenjoyable. Start a new game, or switch to another household, without any Infants. Take a camping trip with Outdoor Retreat, try the Scientist career with Get to Work, go skiing in Snowy Escape, relax on the beach with Island Living, etc. You could even try having the Infants grow up early.
The point being, The Sims 4 has lots of things to do, so when you find that one part of it is unenjoyable, just do something else. I know this doesn't fix the problem but it does make it far less important. I've had problems with knitting, fishing (in Sulani), and high school since the last update but it doesn't really bother me. I've just put those saves on hold while I finished up the Doctor career, started a University degree, had several new romances, and done numerous other things in numerous saves.
I've even been playing one family with twin newborns, although I'm really looking forward to them growing up. I hope I don't find the Infants as frustrating as you have.