Party not joining a game together

Since launch on xbox my brother and I have had to constantly quit out of games because one of us was not put in the game together. This has been extremely frustrating and makes me want to just stop.p...