Party doesn't always load in a match together

My party is usually a party of 3 or 4 with a mix of PC and Xbox. No matter who is in the party or who is the party leader, most of the time only 2 or 3 of us will load in, and the other(s) gets stuck...