Lobby crashes when loading onto next game more on crossplay

I am on xbox series x and play cross play with 2 friends on pc, when doing this it constantly doesn't load me into the game with them, it will go to load the map after matchmaking then goes to load before returning me back to the lobby and the two friends go into the game.

This happens constantly and have to try 5-10 times to play one game.