Games filling and emptying without me

This seems to happen consistently when the player count is low (like early morning 8AMish). Games will appear to fill based on the numbers in the game mode indicator. Game will reach some 60+ number and empty out back to 10. I will still be in the lobby. This will happen on repeat. 7-8 times. It's getting really old.

What the hell is going on?