Xbox infinite loading screen

For some reason when I try to return to lobby after a game I get stuck in an infinite loading screen and I have to restart my game every time it happens. I'm on Xbox.