Empty servers after one round...

Almost every round I play now, one round will be completely full.  After it ends and we should be switching sides, I end up sitting at the screen "Waiting for players 10/64" and it only goes down as ...