Why is the game skipping many hours when restarted?

Were your sims sleeping when you left the game? That happens to me when I leave the game and all my sims are asleep. The game is fast forwarding through the night during the loading screen, so when you get to your household, it's now the morning and your sims are awake. It's super annoying because normally I like to get my sims up at 5 am, but when this happens, they don't get up until 7 am or later since the game skipped all that time.