I have the relationship culling mod so I don't really get that issue. Generally culling I don't have too much issue with in my game as I make my characters and have fewer townies. Again, mod. It did happen though in a main save game of mine.
My elder in my main family works as a scientist and before IRL Christmas, there was one co-worker with last name Quick that worked on the team. I was busy and couldn't play main save over Christmas or New Year's. When I went back in, he had been replaced. Essentially, from the looks of things, the game fired Quick and made his household move out. :D