Re: My infant lives with his grandparents
First of all, sorry to hear you have trouble with the accessibility of the game. I'm glad you've found a workaround that works for you (for the most part).
That said, this happened to me once, and the newborn disappeared from the game. I wasn't playing with the household though (only the father). There isn't an in-game option to then move the baby. Once it is left alone, CPS swoops in (behind the scenes, presumably), and takes the baby away permanently.
For anyone on PC, MCCC (mod) has the ability to have the game reassign a family (I've tested it and had a baby "adopted" by ignoring it for days, lol). For console, however, your best bet is to have a younger sim in the household, at least teen, so that your risk is not so high. I know you are on console, but this is for anyone else who may have the same issue/question who are not on console.
Since you asked only about the grampa and the baby, perhaps move in a cousin/other relative or add a live-in "nanny" (not the game nanny, but just a household sim that you use as an extra caretaker). That would give you three sims in a household, and under your limit of four.
Good luck and happy simming!