Unable to invite friends to rush?!!!

I have series x and friends have Xbox ones/s when some of us try to invite the others to rush it says cannot send invite. Only one of the 4 of us is able to invite? Any suggestions?