Fast forward not working

I was playing the other week when I noticed I can’t fast forward time when someone from outside the household is visiting 

as soon as they leave I can fast forward
it dose happen occasionally when I don’t have visitors but I just come out of the game and reload then it fixes it but it’s more prominent when there’s a visitor 

That's how the game has worked for as long as I remember. My Sims frequently have night time dates, vampire visits, and other late night visitors. The game remains at standard speeds until all the visitors have left or everyone on the lot is sleeping. I can go to speed 3 but "fast forward" only works when there isn't anyone active on the lot.

As I said, that's just normal behavior for the game.