Getting stuck between matches

I've had this for a while now but especially when I'm playing multiplayer (happens especially when I'm in a party - all my friends are cross platform) I end up lagged out in the launch screen. When I'm not party leader this means I'm forced to quit to menu and have no way of joining them or if I'm the leader I have to pull them out of a game and into a new one. Doesn't happen every game but at least every 2-3 if I'm lucky!

I have an Xbox X and a physical disc