Won’t load both players into challenge!

We have been trying to play co-op on Xbox for a couple of days but everytime we try to load into a challenge one player is left out and then the other can’t abandon it or anything. It’s becoming annoying and I’ve already spent a lot of time and energy on the game.