Time when sims sleep

I’ve noticed that I cannot fast forward the time when all of my sims are asleep, I can only put it on the normal play speed, if I try to bump up the speed of the time, it’ll just go back to the first one unless a sim is awake. 
-Xbox