Friends can't follow me game to game

I join a party or they join mine. I am on Steam, some of them have been on Steam or Xbox and when we get into a game it will leave someone in the menu and fill the game without them, causing us to leave so we can pick them up. This is something that also happens when we are just riding out the game session  going game to game it will leave someone behind as well.  It has happened seemingly very randomly but it happens multiple times in a 4 hour gaming session. Please fix!