Games filling and emptying without me

This seems to happen consistently when the player count is low (like early morning 8AMish). Games will appear to fill based on the numbers in the game mode indicator. Game will reach some 60+ number ...