NRAAS rotational play question about butlers

I play rotationally using NRAAS. All of my families have progressed to having large houses with live in butlers. 100% of the time when I rotate back into a family, the butler is gone, the house is a ...