Closed my game without saving and reloaded. I even slowed down time to normal instead of leaving it on 3 (since both parents were sleeping, it automatically kicked it up to ultra speed), and kept an eye on the kid. He wasn't crying or anything, just kicking around happily in the bassinet. As soon as his mum woke up, I sent her to feed him and do all the various socials with him, but before she could even get to the bassinet, I got the notification again: "Due to neglect, Ace is being whisked away to safety. Maybe you should consider a goldfish before jumping into parenthood again."
There should DEFINITELY be a warning first. :(
Now to close without saving and open again ... this time I'll wake the mum up and feed him and play with him before it can take him away.