Can’t start challenges without randoms joining me and crashing my game

every time I start a challenge, it says that there’s more than the current party then once the challenge is finished, my game crashes every single time I start a challenge the only way for me to get around it is by joining one of my friends and then passing me the leader. Once they’ve passed a leader, I don’t seem to have any issues but when creating my own lobby/Party the game seems to add people to my challenges every single time. In other words, it’s game breaking because I cannot be dealing with this far much longer.