Party not ready...?

After everygame I play, either with my buddies or randoms, i have to leave the party and rejoin or go back to the menu or else my party will get put into a game by themselves or if I'm alone it says "your party is not ready" when there is no one else. Generally the only time this doesnt happen is if the game ends quickly, then this problem doenst arise. I know some of my buddy's friends have the same problem.