Party doesn't always load in a match together

My party is usually a party of 3 or 4 with a mix of PC and Xbox. No matter who is in the party or who is the party leader, most of the time only 2 or 3 of us will load in, and the other(s) gets stuck in the main menu. Our in-match squad will have vacancies that are then filled with random players. We end up needing to all go back to the main menu and try matchmaking again. Sometimes it puts us all in a match together immediately, but sometimes it can take 4 or 5 tries to all get in a match together.