I don't always but the family I am currently playing I have. I haven't found it too difficult. I love the trait you get for maxing out their skills as it makes levelling skills up on children so much ...
I do, but I play on long life so I have the time to do it. If I played on Normal I'd probably focus on just getting them to 3, if I worried about them at all.