Don't always load into game when in a party.

I've been noticing that when I try to load in a game with a party sometimes I do not always load in with them. I am playing on PC and so far I've only noticed this while playing with my friends that ...